Jul 06 18:54:56.213896 ip-10-0-138-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 18:54:56.641700 ip-10-0-138-123 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 18:54:56.641700 ip-10-0-138-123 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 18:54:56.641700 ip-10-0-138-123 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 18:54:56.641700 ip-10-0-138-123 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 18:54:56.641700 ip-10-0-138-123 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 18:54:56.643820 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.643723    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 18:54:56.648845 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648825    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.648845 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648845    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648849    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648852    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648855    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648858    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648863    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648865    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648869    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648872    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648875    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648878    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648880    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648883    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648885    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648888    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648891    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648893    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648896    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648900    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648903    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.648914 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648906    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648908    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648912    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648914    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648917    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648920    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648923    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648926    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648930    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648934    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648937    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648940    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648942    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648944    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648947    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648949    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648952    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648955    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648960    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.649449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648964    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648967    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648969    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648973    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648976    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648978    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648981    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648984    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648986    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648989    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648991    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648993    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648996    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.648998    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649001    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649005    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649007    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649010    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649013    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649015    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.649938 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649018    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649020    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649023    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649025    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649028    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649031    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649033    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649035    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649039    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649041    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649043    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649048    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649050    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649053    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649056    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649058    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649061    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649063    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649066    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649069    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.650413 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649071    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649075    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649078    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649081    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649084    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649087    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649089    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649093    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649096    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649099    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649101    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649104    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649107    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649109    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649112    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649114    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649117    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649120    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649122    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649921    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.650911 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649929    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649932    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649935    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649938    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649941    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649944    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649947    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649950    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649953    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649955    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649959    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649964    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649967    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649969    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649972    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649975    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649978    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649981    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649983    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649987    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.651393 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649990    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649993    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649996    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.649998    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650001    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650004    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650006    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650009    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650011    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650014    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650016    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650019    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650021    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650024    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650027    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650030    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650032    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650035    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650038    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.651933 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650041    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650044    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650046    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650049    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650051    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650054    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650056    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650059    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650062    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650064    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650067    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650069    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650073    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650076    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650078    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650081    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650083    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650086    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650090    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650093    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.652411 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650095    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650098    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650100    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650105    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650108    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650111    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650114    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650117    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650119    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650122    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650126    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650136    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650139    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650142    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650144    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650147    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650150    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650153    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650157    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.652922 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650160    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650162    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650165    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650168    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650172    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650174    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650177    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650179    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650182    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650185    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650187    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650190    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650192    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650195    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650197    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650200    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650202    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650205    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650207    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650209    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.653495 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650665    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650670    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650673    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650676    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650679    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650681    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650684    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650687    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650690    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650693    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650697    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650699    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650702    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650705    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650707    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650712    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650715    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650718    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650721    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650724    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.654011 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650727    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650730    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650732    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650735    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650737    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650740    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650743    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650745    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650748    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650750    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650753    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650755    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650758    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650761    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650763    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650766    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650769    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650779    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650782    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650784    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.654491 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650787    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650789    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650794    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650797    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650799    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650802    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650804    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650806    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650809    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650811    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650814    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650818    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650821    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650823    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650826    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650828    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650830    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650833    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650836    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.654985 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650838    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650842    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650846    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650848    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650851    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650854    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650857    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650860    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650863    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650866    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650868    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650876    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650881    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650883    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650886    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650890    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650893    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650895    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650898    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.655449 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650900    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650903    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650906    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650908    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650911    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650913    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650916    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650919    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650922    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650925    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650928    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650932    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650934    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650937    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650939    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650941    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650944    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650946    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650949    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650951    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.650954    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.655917 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651040    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651051    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651068    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651075    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651083    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651088    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651093    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651100    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651103    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651106    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651110    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651113    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651116    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651119    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651122    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651125    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651127    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651130    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651135    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651138    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651141    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651144    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651148    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651152    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 18:54:56.656425 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651155    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651158    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651161    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651164    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651167    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651170    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651173    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651176    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651181    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651184    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651187    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651190    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651193    2561 flags.go:64] FLAG: --enable-server="true"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651196    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651201    2561 flags.go:64] FLAG: --event-burst="100"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651205    2561 flags.go:64] FLAG: --event-qps="50"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651208    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651212    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651215    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651219    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651222    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651225    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651228    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651231    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651234    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 18:54:56.657065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651237    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651239    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651242    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651245    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651248    2561 flags.go:64] FLAG: --feature-gates=""
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651252    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651255    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651258    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651261    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651264    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651267    2561 flags.go:64] FLAG: --help="false"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651270    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651273    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651276    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651279    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651283    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651286    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651289    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651292    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651295    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651298    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651302    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651307    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651310    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 06 18:54:56.657698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651313    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651315    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651319    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651322    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651324    2561 flags.go:64] FLAG: --lock-file=""
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651327    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651330    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651333    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651339    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651342    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651344    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651347    2561 flags.go:64] FLAG: --logging-format="text"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651350    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651353    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651356    2561 flags.go:64] FLAG: --manifest-url=""
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651359    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651363    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651366    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651370    2561 flags.go:64] FLAG: --max-pods="110"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651373    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651376    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651379    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651382    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651385    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651388    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 18:54:56.658293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651391    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651398    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651401    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651404    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651407    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651411    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651417    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651420    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651423    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651426    2561 flags.go:64] FLAG: --port="10250"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651429    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651432    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c9ebd453c84c763f"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651435    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651438    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651441    2561 flags.go:64] FLAG: --register-node="true"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651443    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651447    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651450    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651453    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651455    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651459    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651462    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651465    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651468    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651471    2561 flags.go:64] FLAG: --runonce="false"
Jul 06 18:54:56.658954 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651474    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651477    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651480    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651483    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651486    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651489    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651492    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651495    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651498    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651502    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651505    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651508    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651511    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651515    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651518    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651524    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651527    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651530    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651535    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651538    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651541    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651543    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651546    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651549    2561 flags.go:64] FLAG: --v="2"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651566    2561 flags.go:64] FLAG: --version="false"
Jul 06 18:54:56.659602 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651571    2561 flags.go:64] FLAG: --vmodule=""
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651574    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.651578    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651714    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651718    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651721    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651724    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651727    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651730    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651732    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651735    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651737    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651739    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651742    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651745    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651747    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651755    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651757    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651760    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651763    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.660214 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651765    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651769    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651771    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651774    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651778    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651782    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651785    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651787    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651790    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651793    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651797    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651799    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651802    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651804    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651807    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651810    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651812    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651815    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651817    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.660777 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651820    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651822    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651842    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651846    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651849    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651852    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651855    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651858    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651861    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651867    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651869    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651872    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651874    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651877    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651879    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651882    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651885    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651888    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651890    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651893    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.661242 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651895    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651898    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651900    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651902    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651905    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651908    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651910    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651913    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651915    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651918    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651920    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651922    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651925    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651927    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651930    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651932    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651934    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651937    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651939    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651942    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.661752 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651945    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651948    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651951    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651953    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651956    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651960    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651962    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651965    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651968    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651970    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651973    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651975    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651978    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651981    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651984    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651986    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651988    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651992    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651995    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.651999    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.662229 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.652002    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.652004    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.652007    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.652899    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.660356    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.660374    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.662097    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.662107    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662170    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662176    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662180    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662183    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662186    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662189    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662192    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.662750 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662195    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662199    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662203    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662206    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662210    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662212    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662216    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662219    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662222    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662224    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662227    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662230    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662233    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662235    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662238    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662241    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662246    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662249    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.663122 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662252    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662255    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662258    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662262    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662265    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662267    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662270    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662272    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662275    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662277    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662280    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662284    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662287    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662290    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662293    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662295    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662298    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662300    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662303    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.663616 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662306    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662309    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662312    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662314    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662317    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662320    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662322    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662324    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662327    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662329    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662332    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662334    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662337    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662339    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662342    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662344    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662347    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662351    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662353    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662356    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.664073 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662358    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662361    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662363    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662366    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662368    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662371    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662373    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662376    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662378    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662380    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662383    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662385    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662388    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662390    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662393    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662395    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662398    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662400    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662402    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662405    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.664581 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662407    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662410    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662412    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662415    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662417    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662420    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662422    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662424    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662428    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662430    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662435    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662437    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662440    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662443    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.662448    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 18:54:56.665062 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662615    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662622    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662625    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662627    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662630    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662633    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662635    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662638    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662641    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662643    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662646    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662648    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662651    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662653    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662657    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662661    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662665    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662668    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662671    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 18:54:56.665440 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662673    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662676    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662678    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662681    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662684    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662686    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662688    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662691    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662694    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662701    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662705    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662707    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662711    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662714    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662718    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662721    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662724    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662727    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662729    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 18:54:56.665925 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662732    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662735    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662737    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662740    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662742    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662745    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662747    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662750    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662752    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662755    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662758    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662761    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662764    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662766    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662769    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662771    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662774    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662776    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662779    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662781    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 18:54:56.666436 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662784    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662787    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662789    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662793    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662795    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662797    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662800    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662803    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662805    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662808    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662810    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662813    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662815    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662818    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662820    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662822    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662825    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662827    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662830    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 18:54:56.666956 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662832    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662835    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662837    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662840    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662843    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662845    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662848    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662850    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662852    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662855    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662858    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662862    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662865    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662868    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662870    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662873    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662876    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662879    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662881    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662883    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 18:54:56.667415 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662886    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:56.662889    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.662893    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.663029    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.665925    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.667636    2561 server.go:1019] "Starting client certificate rotation"
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.667735    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 18:54:56.668034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.667807    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 18:54:56.671806 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.671790    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 18:54:56.677227 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.677198    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 18:54:56.688948 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.688916    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 06 18:54:56.691571 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.691536    2561 log.go:25] "Validated CRI v1 image API"
Jul 06 18:54:56.692510 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.692484    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 18:54:56.694459 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.694442    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 18:54:56.695293 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.695268    2561 fs.go:135] Filesystem UUIDs: map[3ec32de3-f720-4465-8f9d-8e73168acd15:/dev/nvme0n1p3 74f48e7f-f8f2-4bf8-b9fb-628db384a7a5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 06 18:54:56.695381 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.695292    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 18:54:56.700693 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.700543    2561 manager.go:216] Machine: {Timestamp:2026-07-06 18:54:56.699118049 +0000 UTC m=+0.368229691 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099609 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29da08eb1d1bc62ec12c5cbfdb401d SystemUUID:ec29da08-eb1d-1bc6-2ec1-2c5cbfdb401d BootID:5c114565-c9a1-4367-9578-f603998ff726 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b2:18:01:e9:f5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b2:18:01:e9:f5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:be:f0:8c:66:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 18:54:56.700693 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.700677    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 18:54:56.700868 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.700801    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 18:54:56.704496 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.704464    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 18:54:56.704660 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.704501    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 18:54:56.705382 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.705372    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 18:54:56.705416 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.705385    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 18:54:56.705416 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.705394    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 18:54:56.705416 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.705408    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 18:54:56.707323 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.707311    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 18:54:56.711913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.711898    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 18:54:56.712034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712024    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 18:54:56.712068 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712055    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 18:54:56.712104 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712075    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 18:54:56.712145 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712106    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 18:54:56.712145 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712116    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 06 18:54:56.712145 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712126    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 18:54:56.712581 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712545    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 18:54:56.712615 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.712586    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 18:54:56.715064 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.715042    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 18:54:56.715179 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.715153    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gsrbt"
Jul 06 18:54:56.717583 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.717549    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 18:54:56.717667 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.717599    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 18:54:56.719026 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719013    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 18:54:56.719072 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719036    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 18:54:56.719072 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719043    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 18:54:56.719072 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719050    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 18:54:56.719072 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719060    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 18:54:56.719072 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719067    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719076    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719083    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719103    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719111    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719119    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 18:54:56.719208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.719125    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 18:54:56.720059 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.720046    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 18:54:56.720059 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.720059    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 18:54:56.723906 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.723887    2561 server.go:1268] "Started kubelet"
Jul 06 18:54:56.725026 ip-10-0-138-123 systemd[1]: Started Kubernetes Kubelet.
Jul 06 18:54:56.725587 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.725025    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 18:54:56.725587 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.725079    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 18:54:56.725587 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.725295    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 18:54:56.725587 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.725478    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gsrbt"
Jul 06 18:54:56.725935 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.725912    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 18:54:56.726232 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.726121    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 18:54:56.726287 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.726236    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 18:54:56.727936 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.727918    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 18:54:56.729042 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.729025    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 18:54:56.730715 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.730683    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 18:54:56.736165 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.734976    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-123.ec2.internal.18bfc8328268d3b9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-123.ec2.internal,UID:ip-10-0-138-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-123.ec2.internal,},FirstTimestamp:2026-07-06 18:54:56.723833785 +0000 UTC m=+0.392945428,LastTimestamp:2026-07-06 18:54:56.723833785 +0000 UTC m=+0.392945428,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-123.ec2.internal,}"
Jul 06 18:54:56.737214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.737198    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 18:54:56.737214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.737205    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 18:54:56.737664 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.737426    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:56.738113 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738072    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 18:54:56.738113 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738088    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 18:54:56.738370 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738212    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 18:54:56.738370 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738224    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 18:54:56.738608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738525    2561 factory.go:55] Registering systemd factory
Jul 06 18:54:56.738608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738542    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 06 18:54:56.738608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.738579    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 18:54:56.739030 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739015    2561 factory.go:153] Registering CRI-O factory
Jul 06 18:54:56.739089 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739032    2561 factory.go:223] Registration of the crio container factory successfully
Jul 06 18:54:56.739089 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739083    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 18:54:56.739184 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739103    2561 factory.go:103] Registering Raw factory
Jul 06 18:54:56.739184 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739114    2561 manager.go:1196] Started watching for new ooms in manager
Jul 06 18:54:56.739763 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.739739    2561 manager.go:349] Starting recovery of all containers
Jul 06 18:54:56.748313 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.748289    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:54:56.751890 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.751708    2561 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.752910 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.752894    2561 manager.go:354] Recovery completed
Jul 06 18:54:56.757694 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.757680    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.759573 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.759542    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.759626 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.759588    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.759626 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.759603    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.760241 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.760219    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 18:54:56.760241 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.760241    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 18:54:56.760372 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.760262    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 18:54:56.760372 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.760224    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.761710 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.761694    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.761804 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.761730    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.761804 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.761745    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.762408 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.762394    2561 policy_none.go:49] "None policy: Start"
Jul 06 18:54:56.762472 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.762413    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 18:54:56.762472 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.762427    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 18:54:56.764518 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.764503    2561 policy_none.go:47] "Start"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814329    2561 manager.go:337] "Starting Device Plugin manager"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.814372    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814382    2561 server.go:87] "Starting device plugin registration server"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814722    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814738    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814841    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814946    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.814959    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.815586    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 18:54:56.819502 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.815634    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:56.871572 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.871482    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 18:54:56.872859 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.872842    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 18:54:56.872936 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.872881    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 18:54:56.872936 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.872906    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 18:54:56.873028 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.872949    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 18:54:56.875477 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.875450    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:54:56.915355 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.915319    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.916358 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.916337    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.916485 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.916369    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.916485 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.916380    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.916485 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.916405    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.925056 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.925035    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.925180 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.925065    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-123.ec2.internal\": node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:56.942196 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:56.942159    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:56.973269 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.973215    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"]
Jul 06 18:54:56.973397 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.973344    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.975294 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.975278    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.975389 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.975313    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.975389 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.975328    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.977810 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.977793    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.977926 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.977911    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.977975 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.977941    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.978637 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978619    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.978718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978621    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.978718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978684    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.978718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978695    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.978718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978657    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.978718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.978720    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:56.981057 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.981042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:56.981115 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.981069    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 18:54:56.981887 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.981868    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 18:54:56.981976 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.981899    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 18:54:56.981976 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:56.981909    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeHasSufficientPID"
Jul 06 18:54:57.023710 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.023678    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.027335 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.027316    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-123.ec2.internal\" not found" node="ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.039694 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.039667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.039812 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.039698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.039812 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.039715    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7389aae70d56334e642e67fe3222430-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"a7389aae70d56334e642e67fe3222430\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.042914 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.042895    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.140100 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.140100 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.140287 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140105    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.140287 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140122    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/93b307a79aae92b4bba02d5170237696-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal\" (UID: \"93b307a79aae92b4bba02d5170237696\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.140287 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7389aae70d56334e642e67fe3222430-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"a7389aae70d56334e642e67fe3222430\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.140287 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.140167    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a7389aae70d56334e642e67fe3222430-config\") pod \"kube-apiserver-proxy-ip-10-0-138-123.ec2.internal\" (UID: \"a7389aae70d56334e642e67fe3222430\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.143116 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.143095    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.243991 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.243949    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.326130 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.326087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.329908 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.329882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.344507 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.344479    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.445169 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.445072    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.545711 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.545668    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.646316 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.646281    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.666486 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.666450    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 18:54:57.666651 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.666634    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 18:54:57.666690 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.666642    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 18:54:57.734387 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.734293    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 18:49:57 +0000 UTC" deadline="2027-12-26 11:20:47.8889021 +0000 UTC"
Jul 06 18:54:57.734387 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.734339    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12904h25m50.154566749s"
Jul 06 18:54:57.747143 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.747109    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.752518 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.752495    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:54:57.848121 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:57.848089    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-123.ec2.internal\" not found"
Jul 06 18:54:57.894098 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:57.894048    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b307a79aae92b4bba02d5170237696.slice/crio-1600cc7ec460683273ac830355672413e1296e3bcd7b9d9a487eaa02e19868b3 WatchSource:0}: Error finding container 1600cc7ec460683273ac830355672413e1296e3bcd7b9d9a487eaa02e19868b3: Status 404 returned error can't find the container with id 1600cc7ec460683273ac830355672413e1296e3bcd7b9d9a487eaa02e19868b3
Jul 06 18:54:57.894334 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:54:57.894312    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7389aae70d56334e642e67fe3222430.slice/crio-7933ab879341a97020ac63678bb3c60280bff30b3cb6fd59d1ad17e1b1f2157b WatchSource:0}: Error finding container 7933ab879341a97020ac63678bb3c60280bff30b3cb6fd59d1ad17e1b1f2157b: Status 404 returned error can't find the container with id 7933ab879341a97020ac63678bb3c60280bff30b3cb6fd59d1ad17e1b1f2157b
Jul 06 18:54:57.897871 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.897856    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 18:54:57.902475 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.902458    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:54:57.938271 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.938236    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.949128 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.949101    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 18:54:57.949957 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.949938    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal"
Jul 06 18:54:57.958539 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:57.958522    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 18:54:59.796913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.796826    2561 apiserver.go:52] "Watching apiserver"
Jul 06 18:54:59.806335 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.806303    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 18:54:59.807986 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.807946    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ml7kg","openshift-network-diagnostics/network-check-target-2w7v7","openshift-cluster-node-tuning-operator/tuned-sh27w","openshift-image-registry/node-ca-j4jpp","openshift-multus/multus-additional-cni-plugins-dlp6k","openshift-multus/network-metrics-daemon-6f5c8","openshift-network-operator/iptables-alerter-7jrxx","openshift-ovn-kubernetes/ovnkube-node-fgzhm","kube-system/global-pull-secret-syncer-56wjc","kube-system/konnectivity-agent-zch6z","kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk","openshift-dns/node-resolver-q95qr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal"]
Jul 06 18:54:59.810435 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.810413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.812959 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.812934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.813172 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.813157    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.813404 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.813390    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdvsd\""
Jul 06 18:54:59.813671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.813655    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 18:54:59.814208 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.814186    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.814974 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.814954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.816212 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.816183    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.816705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.816688    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.816790 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.816696    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jgkxq\""
Jul 06 18:54:59.817429 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.817056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.817429 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.817115    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.817429 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.817222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-flw4g\""
Jul 06 18:54:59.817683 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.817438    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.819349 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.819329    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 18:54:59.819663 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.819644    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.819663 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.819660    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-75r2r\""
Jul 06 18:54:59.819789 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.819749    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.820591 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.820571    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.821091 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.821074    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 18:54:59.822876 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.822783    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 18:54:59.823581 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.823518    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.823671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.823583    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nx285\""
Jul 06 18:54:59.823671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.823595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 18:54:59.823671 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:59.823593    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:54:59.823671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.823609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 18:54:59.823671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.823652    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.824046 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.824030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.827105 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.827087    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:54:59.827214 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:59.827173    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:54:59.832283 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832262    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.832633 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832593    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-hosts-file\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.832751 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-tmp-dir\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.832751 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysconfig\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.832751 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832695    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-conf\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.832751 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-lib-modules\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-var-lib-kubelet\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-serviceca\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmz7l\" (UniqueName: \"kubernetes.io/projected/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-kube-api-access-wmz7l\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832851    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz864\" (UniqueName: \"kubernetes.io/projected/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-kube-api-access-mz864\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-modprobe-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.832960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrdmt\" (UniqueName: \"kubernetes.io/projected/fe78c48b-7177-4cc0-908d-fbe69ed06e12-kube-api-access-wrdmt\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.832989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-system-cni-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5771f050-65b3-425d-9807-049c2b94d769-iptables-alerter-script\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77zbk\" (UniqueName: \"kubernetes.io/projected/5771f050-65b3-425d-9807-049c2b94d769-kube-api-access-77zbk\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-kubernetes\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-tuned\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-cnibin\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmz7\" (UniqueName: \"kubernetes.io/projected/b28c3344-93f0-4722-b2dd-c3a0e258839f-kube-api-access-hdmz7\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-sys\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-tmp\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833249 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833259    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5771f050-65b3-425d-9807-049c2b94d769-host-slash\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-host\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833266    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-os-release\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833440    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-systemd\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833531    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-host\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833556    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnh7\" (UniqueName: \"kubernetes.io/projected/52bea5b7-0315-4919-beb2-4954ccfea43a-kube-api-access-tvnh7\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-run\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.833608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.833601    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.834683 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.834416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4lq47\""
Jul 06 18:54:59.834683 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.834544    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 18:54:59.834922 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.834902    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:54:59.835029 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:59.835005    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:54:59.837365 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.837340    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:54:59.839432 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.839412    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 18:54:59.840442 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.839571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2bzrn\""
Jul 06 18:54:59.840593 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.839715    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 18:54:59.840698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.839948    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.842802 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.842782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.843125 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.843107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rbhcb\""
Jul 06 18:54:59.843244 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.842928    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 18:54:59.843584 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.842789    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 18:54:59.843910 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.843885    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 18:54:59.844062 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.844045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.844214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.844196    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 18:54:59.844421 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.844403    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.845809 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.845673    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zdq77\""
Jul 06 18:54:59.845809 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.845742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 18:54:59.845921 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.845855    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 18:54:59.846486 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.846437    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 18:54:59.852433 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.852405    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xs5z6"
Jul 06 18:54:59.859983 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.859960    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xs5z6"
Jul 06 18:54:59.922539 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.922509    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 18:54:59.933825 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933790    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-sys\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5771f050-65b3-425d-9807-049c2b94d769-host-slash\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-ovn\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-socket-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-etc-kubernetes\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933943    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-sys\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.933966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-host\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934012    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-os-release\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934015    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5771f050-65b3-425d-9807-049c2b94d769-host-slash\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-systemd\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934021    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-host\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934087    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-os-release\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnh7\" (UniqueName: \"kubernetes.io/projected/52bea5b7-0315-4919-beb2-4954ccfea43a-kube-api-access-tvnh7\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-systemd\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8485ac65-9021-4ac1-97fb-cdc1bc397845-agent-certs\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-etc-selinux\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934205    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dd7g\" (UniqueName: \"kubernetes.io/projected/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kube-api-access-5dd7g\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8s2g\" (UniqueName: \"kubernetes.io/projected/ea88e9c0-af09-433a-a05b-bafaf2faacaf-kube-api-access-k8s2g\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-run\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-registration-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934343 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-netns\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934367    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-run\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934373    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-var-lib-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-device-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-conf-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934445    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934484    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-conf\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-lib-modules\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-serviceca\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934573    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-system-cni-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934600    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-lib-modules\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934599    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5771f050-65b3-425d-9807-049c2b94d769-iptables-alerter-script\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934647    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-conf\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934651    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-system-cni-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8485ac65-9021-4ac1-97fb-cdc1bc397845-konnectivity-ca\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-bin\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934727    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrdmt\" (UniqueName: \"kubernetes.io/projected/fe78c48b-7177-4cc0-908d-fbe69ed06e12-kube-api-access-wrdmt\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.934909 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77zbk\" (UniqueName: \"kubernetes.io/projected/5771f050-65b3-425d-9807-049c2b94d769-kube-api-access-77zbk\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934776    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-netd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934800    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-socket-dir-parent\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-multus-certs\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-kubernetes\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-tuned\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-systemd-units\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-config\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-dbus\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934964    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-serviceca\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.934985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-bin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-tmp\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935037    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-kubernetes\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935059    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935076    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5771f050-65b3-425d-9807-049c2b94d769-iptables-alerter-script\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935107    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-kubelet\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.935767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c36617-d4dc-47f9-a660-607fff2508ab-ovn-node-metrics-cert\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cni-binary-copy\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-hostroot\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-daemon-config\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935273    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-sys-fs\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-host\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935346    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-host\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-systemd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:59.935368    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935353    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935390    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-etc-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:54:59.935441    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:00.435420193 +0000 UTC m=+3.021628165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-os-release\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.936608 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-slash\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935590    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-system-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935641    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/52bea5b7-0315-4919-beb2-4954ccfea43a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgvk\" (UniqueName: \"kubernetes.io/projected/d8c36617-d4dc-47f9-a660-607fff2508ab-kube-api-access-cbgvk\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935769    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-hosts-file\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-tmp-dir\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysconfig\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935857    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-hosts-file\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-var-lib-kubelet\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmz7l\" (UniqueName: \"kubernetes.io/projected/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-kube-api-access-wmz7l\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935916    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysconfig\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935926    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-var-lib-kubelet\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937367 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-kubelet-config\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-k8s-cni-cncf-io\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-kubelet\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.935994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz864\" (UniqueName: \"kubernetes.io/projected/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-kube-api-access-mz864\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936085    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-tmp-dir\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-modprobe-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936142    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-netns\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-log-socket\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936254    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-modprobe-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936261    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-sysctl-d\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cnibin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-multus\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936322    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-cnibin\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936350    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmz7\" (UniqueName: \"kubernetes.io/projected/b28c3344-93f0-4722-b2dd-c3a0e258839f-kube-api-access-hdmz7\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936549    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/52bea5b7-0315-4919-beb2-4954ccfea43a-cnibin\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.937958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936524    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-node-log\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.938559 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-env-overrides\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.938559 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.936605    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-script-lib\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:54:59.938795 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.938771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-tmp\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.942214 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.942186    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fe78c48b-7177-4cc0-908d-fbe69ed06e12-etc-tuned\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.945454 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945430    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz864\" (UniqueName: \"kubernetes.io/projected/cc8e9d9f-43be-4e26-b5f6-04e8177a4d47-kube-api-access-mz864\") pod \"node-resolver-q95qr\" (UID: \"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47\") " pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:54:59.945595 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945439    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmz7l\" (UniqueName: \"kubernetes.io/projected/fcc43497-d4f3-4732-b4ab-a3c351f6b64e-kube-api-access-wmz7l\") pod \"node-ca-j4jpp\" (UID: \"fcc43497-d4f3-4732-b4ab-a3c351f6b64e\") " pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:54:59.945705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945661    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrdmt\" (UniqueName: \"kubernetes.io/projected/fe78c48b-7177-4cc0-908d-fbe69ed06e12-kube-api-access-wrdmt\") pod \"tuned-sh27w\" (UID: \"fe78c48b-7177-4cc0-908d-fbe69ed06e12\") " pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:54:59.945811 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945745    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmz7\" (UniqueName: \"kubernetes.io/projected/b28c3344-93f0-4722-b2dd-c3a0e258839f-kube-api-access-hdmz7\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:54:59.945811 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945762    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77zbk\" (UniqueName: \"kubernetes.io/projected/5771f050-65b3-425d-9807-049c2b94d769-kube-api-access-77zbk\") pod \"iptables-alerter-7jrxx\" (UID: \"5771f050-65b3-425d-9807-049c2b94d769\") " pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:54:59.945899 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.945828    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnh7\" (UniqueName: \"kubernetes.io/projected/52bea5b7-0315-4919-beb2-4954ccfea43a-kube-api-access-tvnh7\") pod \"multus-additional-cni-plugins-dlp6k\" (UID: \"52bea5b7-0315-4919-beb2-4954ccfea43a\") " pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:54:59.960134 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.960082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"a7389aae70d56334e642e67fe3222430","Type":"ContainerStarted","Data":"7933ab879341a97020ac63678bb3c60280bff30b3cb6fd59d1ad17e1b1f2157b"}
Jul 06 18:54:59.961226 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:54:59.961199    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"93b307a79aae92b4bba02d5170237696","Type":"ContainerStarted","Data":"1600cc7ec460683273ac830355672413e1296e3bcd7b9d9a487eaa02e19868b3"}
Jul 06 18:55:00.036953 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.036912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8485ac65-9021-4ac1-97fb-cdc1bc397845-agent-certs\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:00.036953 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.036954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-etc-selinux\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.036977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dd7g\" (UniqueName: \"kubernetes.io/projected/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kube-api-access-5dd7g\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8s2g\" (UniqueName: \"kubernetes.io/projected/ea88e9c0-af09-433a-a05b-bafaf2faacaf-kube-api-access-k8s2g\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037132    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-etc-selinux\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037153    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-registration-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037216 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-netns\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-var-lib-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-device-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037268    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-registration-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037272    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-conf-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037307    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-var-lib-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037315    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-device-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037223    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037314    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-netns\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037398    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-conf-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037494    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8485ac65-9021-4ac1-97fb-cdc1bc397845-konnectivity-ca\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-bin\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-netd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-socket-dir-parent\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-multus-certs\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-systemd-units\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037697    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-config\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037721    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-dbus\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037711    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-multus-certs\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037670    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-bin\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037745    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-bin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037776    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-systemd-units\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037786    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-bin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-kubelet\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037810    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-kubelet\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c36617-d4dc-47f9-a660-607fff2508ab-ovn-node-metrics-cert\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cni-binary-copy\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.037901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037861    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-cni-netd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-hostroot\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037899    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-daemon-config\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-sys-fs\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-systemd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037969    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-hostroot\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037986    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-etc-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038009    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038056    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8485ac65-9021-4ac1-97fb-cdc1bc397845-konnectivity-ca\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038123    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-os-release\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-os-release\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-slash\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-system-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038213    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038237    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgvk\" (UniqueName: \"kubernetes.io/projected/d8c36617-d4dc-47f9-a660-607fff2508ab-kube-api-access-cbgvk\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.038705 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-kubelet-config\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-k8s-cni-cncf-io\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038385    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-config\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-kubelet\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038449    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-systemd\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038483    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-netns\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.037937    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-dbus\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038484    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cni-binary-copy\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038521    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-log-socket\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038526    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-sys-fs\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-slash\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cnibin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038567    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-multus\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038590    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038522    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-daemon-config\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038616    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.039603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038624    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-run-k8s-cni-cncf-io\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-etc-openvswitch\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038645    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-node-log\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038670    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-env-overrides\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038678    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-netns\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038627    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-log-socket\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038696    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-script-lib\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038700    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-kubelet\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038724    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-node-log\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038591    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/093150fe-2f19-46de-a8e3-714e7ee0779c-kubelet-config\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-ovn\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038757    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-run-ovn\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-socket-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038671    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8c36617-d4dc-47f9-a660-607fff2508ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-etc-kubernetes\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.038823    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038856    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-etc-kubernetes\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.040423 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038876    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-cnibin\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.038892    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:00.538873922 +0000 UTC m=+3.125081893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038893    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-host-var-lib-cni-multus\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.038771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-system-cni-dir\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.039107    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-env-overrides\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.039143    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/49d444ff-acd6-4036-8914-1cd3cbfe8edd-socket-dir\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.039313    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8c36617-d4dc-47f9-a660-607fff2508ab-ovnkube-script-lib\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.039400    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ea88e9c0-af09-433a-a05b-bafaf2faacaf-multus-socket-dir-parent\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.039672    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8485ac65-9021-4ac1-97fb-cdc1bc397845-agent-certs\") pod \"konnectivity-agent-zch6z\" (UID: \"8485ac65-9021-4ac1-97fb-cdc1bc397845\") " pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:00.041000 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.040575    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8c36617-d4dc-47f9-a660-607fff2508ab-ovn-node-metrics-cert\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.045497 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.045474    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8s2g\" (UniqueName: \"kubernetes.io/projected/ea88e9c0-af09-433a-a05b-bafaf2faacaf-kube-api-access-k8s2g\") pod \"multus-ml7kg\" (UID: \"ea88e9c0-af09-433a-a05b-bafaf2faacaf\") " pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.045649 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.045623    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dd7g\" (UniqueName: \"kubernetes.io/projected/49d444ff-acd6-4036-8914-1cd3cbfe8edd-kube-api-access-5dd7g\") pod \"aws-ebs-csi-driver-node-rblrk\" (UID: \"49d444ff-acd6-4036-8914-1cd3cbfe8edd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.046769 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.046751    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:00.046769 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.046772    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:00.046960 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.046786    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:00.046960 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.046865    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:00.546848161 +0000 UTC m=+3.133056131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:00.049302 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.049282    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgvk\" (UniqueName: \"kubernetes.io/projected/d8c36617-d4dc-47f9-a660-607fff2508ab-kube-api-access-cbgvk\") pod \"ovnkube-node-fgzhm\" (UID: \"d8c36617-d4dc-47f9-a660-607fff2508ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.132764 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.132723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7jrxx"
Jul 06 18:55:00.140410 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.140373    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q95qr"
Jul 06 18:55:00.151379 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.151344    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ml7kg"
Jul 06 18:55:00.157270 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.157243    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sh27w"
Jul 06 18:55:00.162297 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.162272    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j4jpp"
Jul 06 18:55:00.169065 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.169041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dlp6k"
Jul 06 18:55:00.176719 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.176696    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:00.184633 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.184607    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:00.189567 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.189538    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk"
Jul 06 18:55:00.196984 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.196962    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:55:00.220130 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.220094    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 18:55:00.442147 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.442060    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:00.442342 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.442244    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:00.442342 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.442342    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:01.442318229 +0000 UTC m=+4.028526238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:00.542897 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.542861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:00.543090 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.543028    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:00.543164 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.543113    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:01.543092961 +0000 UTC m=+4.129300924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:00.643761 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.643720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:00.643944 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.643915    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:00.644067 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.643951    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:00.644067 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.643966    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:00.644067 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:00.644025    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:01.644006678 +0000 UTC m=+4.230214630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:00.862663 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.862605    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 18:49:59 +0000 UTC" deadline="2027-12-19 03:01:26.85619359 +0000 UTC"
Jul 06 18:55:00.862663 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.862660    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12728h6m25.993537236s"
Jul 06 18:55:00.870980 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.870940    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5771f050_65b3_425d_9807_049c2b94d769.slice/crio-5f486788ee3f372141cb03980975467ead41d99621eb61e17a174ac74efe89cf WatchSource:0}: Error finding container 5f486788ee3f372141cb03980975467ead41d99621eb61e17a174ac74efe89cf: Status 404 returned error can't find the container with id 5f486788ee3f372141cb03980975467ead41d99621eb61e17a174ac74efe89cf
Jul 06 18:55:00.872003 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.871962    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8e9d9f_43be_4e26_b5f6_04e8177a4d47.slice/crio-0275d13a72fe44447beccdce36c3b6806eebcc121ed69e4686cdc8d538a43df7 WatchSource:0}: Error finding container 0275d13a72fe44447beccdce36c3b6806eebcc121ed69e4686cdc8d538a43df7: Status 404 returned error can't find the container with id 0275d13a72fe44447beccdce36c3b6806eebcc121ed69e4686cdc8d538a43df7
Jul 06 18:55:00.873326 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.873296    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8485ac65_9021_4ac1_97fb_cdc1bc397845.slice/crio-99787499de3ca8951c7567a33cdb757bba56c0dfa16dbe563ca614953c5be3f1 WatchSource:0}: Error finding container 99787499de3ca8951c7567a33cdb757bba56c0dfa16dbe563ca614953c5be3f1: Status 404 returned error can't find the container with id 99787499de3ca8951c7567a33cdb757bba56c0dfa16dbe563ca614953c5be3f1
Jul 06 18:55:00.875964 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.874345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea88e9c0_af09_433a_a05b_bafaf2faacaf.slice/crio-58aaf1ea492d4df80fc04cded6e9d31ea9d83c22e09a18382483a12d34ee8ada WatchSource:0}: Error finding container 58aaf1ea492d4df80fc04cded6e9d31ea9d83c22e09a18382483a12d34ee8ada: Status 404 returned error can't find the container with id 58aaf1ea492d4df80fc04cded6e9d31ea9d83c22e09a18382483a12d34ee8ada
Jul 06 18:55:00.877669 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.877597    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c36617_d4dc_47f9_a660_607fff2508ab.slice/crio-9c51395a6786b0668ec43ee93ae11194a24ab65cece8f6a24c91f8ffee8a956c WatchSource:0}: Error finding container 9c51395a6786b0668ec43ee93ae11194a24ab65cece8f6a24c91f8ffee8a956c: Status 404 returned error can't find the container with id 9c51395a6786b0668ec43ee93ae11194a24ab65cece8f6a24c91f8ffee8a956c
Jul 06 18:55:00.879725 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.879595    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bea5b7_0315_4919_beb2_4954ccfea43a.slice/crio-377264c3c6b158446ba30ec2543a93c43d703c9f09b3b553352acd964e124995 WatchSource:0}: Error finding container 377264c3c6b158446ba30ec2543a93c43d703c9f09b3b553352acd964e124995: Status 404 returned error can't find the container with id 377264c3c6b158446ba30ec2543a93c43d703c9f09b3b553352acd964e124995
Jul 06 18:55:00.880374 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:00.880344    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe78c48b_7177_4cc0_908d_fbe69ed06e12.slice/crio-52ff2376f88066477d82a33192fb76451ea98d88a4362dbb98eff3a4a9d60bd8 WatchSource:0}: Error finding container 52ff2376f88066477d82a33192fb76451ea98d88a4362dbb98eff3a4a9d60bd8: Status 404 returned error can't find the container with id 52ff2376f88066477d82a33192fb76451ea98d88a4362dbb98eff3a4a9d60bd8
Jul 06 18:55:00.964384 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.964191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j4jpp" event={"ID":"fcc43497-d4f3-4732-b4ab-a3c351f6b64e","Type":"ContainerStarted","Data":"43aa55197e6d1e2c83baecc78f0fe8f8dda3bd50c4df69d21f9472326e81a604"}
Jul 06 18:55:00.965514 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.965488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk" event={"ID":"49d444ff-acd6-4036-8914-1cd3cbfe8edd","Type":"ContainerStarted","Data":"fdacb3f34e245b9800a5d075555ba80be33573d2a5e4fcfb43230b73b045a4a5"}
Jul 06 18:55:00.966579 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.966550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sh27w" event={"ID":"fe78c48b-7177-4cc0-908d-fbe69ed06e12","Type":"ContainerStarted","Data":"52ff2376f88066477d82a33192fb76451ea98d88a4362dbb98eff3a4a9d60bd8"}
Jul 06 18:55:00.967676 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.967652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q95qr" event={"ID":"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47","Type":"ContainerStarted","Data":"0275d13a72fe44447beccdce36c3b6806eebcc121ed69e4686cdc8d538a43df7"}
Jul 06 18:55:00.968744 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.968721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ml7kg" event={"ID":"ea88e9c0-af09-433a-a05b-bafaf2faacaf","Type":"ContainerStarted","Data":"58aaf1ea492d4df80fc04cded6e9d31ea9d83c22e09a18382483a12d34ee8ada"}
Jul 06 18:55:00.969789 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.969770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zch6z" event={"ID":"8485ac65-9021-4ac1-97fb-cdc1bc397845","Type":"ContainerStarted","Data":"99787499de3ca8951c7567a33cdb757bba56c0dfa16dbe563ca614953c5be3f1"}
Jul 06 18:55:00.970719 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.970696    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jrxx" event={"ID":"5771f050-65b3-425d-9807-049c2b94d769","Type":"ContainerStarted","Data":"5f486788ee3f372141cb03980975467ead41d99621eb61e17a174ac74efe89cf"}
Jul 06 18:55:00.974125 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.974093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerStarted","Data":"377264c3c6b158446ba30ec2543a93c43d703c9f09b3b553352acd964e124995"}
Jul 06 18:55:00.975163 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.975143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"9c51395a6786b0668ec43ee93ae11194a24ab65cece8f6a24c91f8ffee8a956c"}
Jul 06 18:55:00.987200 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:00.987149    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" podStartSLOduration=1.987124538 podStartE2EDuration="1.987124538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:00.986868688 +0000 UTC m=+3.573076673" watchObservedRunningTime="2026-07-06 18:55:00.987124538 +0000 UTC m=+3.573332512"
Jul 06 18:55:01.450660 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.449988    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:01.450660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.450137    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:01.450660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.450222    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:03.450186299 +0000 UTC m=+6.036394274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:01.556009 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.555971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:01.556228 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.556195    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:01.556304 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.556261    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:03.556241867 +0000 UTC m=+6.142449832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:01.656426 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.656387    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:01.656654 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.656636    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:01.656714 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.656664    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:01.656714 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.656679    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:01.656832 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.656740    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:03.656721063 +0000 UTC m=+6.242929036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:01.863180 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.863043    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 18:49:59 +0000 UTC" deadline="2027-12-06 15:00:14.150381507 +0000 UTC"
Jul 06 18:55:01.863180 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.863090    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12428h5m12.287298563s"
Jul 06 18:55:01.964785 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.964754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:01.964937 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.964873    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:01.964937 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.964887    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:01.965067 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.965007    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:01.965120 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.965069    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:01.965189 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:01.965170    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:01.984757 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:01.984717    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-123.ec2.internal" event={"ID":"a7389aae70d56334e642e67fe3222430","Type":"ContainerStarted","Data":"2ba0d1dd8250f0487248deb93e0523576829187815083a560a1ab59a21663f5c"}
Jul 06 18:55:02.012066 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:02.012019    2561 generic.go:356] "Generic (PLEG): container finished" podID="93b307a79aae92b4bba02d5170237696" containerID="014238f536ec803c81d10cecda66723df5366ec8ece2826e0adfe8fab53ec0c2" exitCode=0
Jul 06 18:55:02.012255 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:02.012088    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"93b307a79aae92b4bba02d5170237696","Type":"ContainerDied","Data":"014238f536ec803c81d10cecda66723df5366ec8ece2826e0adfe8fab53ec0c2"}
Jul 06 18:55:03.029981 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.029915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" event={"ID":"93b307a79aae92b4bba02d5170237696","Type":"ContainerStarted","Data":"aaa26ada84ef5d6f54d69853afdead30994be43a9822fc92f31a9fa6293c0739"}
Jul 06 18:55:03.475568 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.475452    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:03.475730 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.475672    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:03.475802 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.475738    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:07.475719164 +0000 UTC m=+10.061927132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:03.576619 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.576577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:03.576846 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.576828    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:03.576915 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.576899    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:07.576880292 +0000 UTC m=+10.163088249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:03.677302 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.677258    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:03.677520 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.677443    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:03.677520 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.677481    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:03.677520 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.677495    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:03.677684 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.677554    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:07.677534898 +0000 UTC m=+10.263742871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:03.956275 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.956188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:03.956493 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.956450    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:03.956901 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.956882    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:03.957029 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.957005    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:03.957093 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:03.957071    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:03.957159 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:03.957139    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:05.959438 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:05.959407    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:05.959830 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:05.959552    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:05.959972 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:05.959880    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:05.959972 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:05.959954    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:05.960089 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:05.959996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:05.960089 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:05.960038    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:07.509958 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.509819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:07.510421 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.509983    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:07.510421 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.510056    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:15.510037406 +0000 UTC m=+18.096245377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:07.610875 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.610831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:07.611093 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.611074    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:07.611168 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.611159    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:15.611138907 +0000 UTC m=+18.197346862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:07.711843 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.711803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:07.712017 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.712001    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:07.712088 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.712022    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:07.712088 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.712037    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:07.712174 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.712101    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:15.712081801 +0000 UTC m=+18.298289776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:07.960170 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.960096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:07.960326 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.960097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:07.960326 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.960232    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:07.960450 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.960401    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:07.960450 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:07.960098    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:07.960581 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:07.960526    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:09.956775 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:09.956744    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:09.957248 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:09.956864    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:09.957248 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:09.957178    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:09.957356 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:09.957250    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:09.957415 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:09.957368    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:09.957492 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:09.957430    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:11.956556 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:11.956510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:11.956998 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:11.956510    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:11.956998 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:11.956650    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:11.956998 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:11.956670    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:11.956998 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:11.956767    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:11.956998 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:11.956836    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:13.956193 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:13.956158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:13.956681 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:13.956199    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:13.956681 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:13.956160    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:13.956681 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:13.956299    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:13.956681 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:13.956371    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:13.956681 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:13.956480    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:15.572473 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.572411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:15.572951 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.572563    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:15.572951 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.572635    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:55:31.572617957 +0000 UTC m=+34.158825908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:15.673668 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.673628    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:15.673809 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.673789    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:15.673869 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.673855    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:31.673840315 +0000 UTC m=+34.260048269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:15.774791 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.774705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:15.774947 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.774875    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:15.774947 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.774896    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:15.774947 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.774907    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:15.775048 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.774962    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:31.774947255 +0000 UTC m=+34.361155206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:15.959179 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.959149    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:15.959364 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.959248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:15.959364 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.959254    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:15.959364 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:15.959285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:15.959528 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.959390    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:15.959579 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:15.959532    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:17.956959 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:17.956917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:17.957435 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:17.957034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:17.957435 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:17.957154    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:17.957435 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:17.957041    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:17.957435 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:17.957211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:17.957435 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:17.957316    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:19.956785 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:19.956747    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:19.957260 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:19.956787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:19.957260 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:19.956879    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:19.957260 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:19.956919    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:19.957260 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:19.957016    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:19.957260 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:19.957063    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:21.067861 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.067494    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="4ffc68879924cf7df306a59374b206447037614e22bd7f4b37ba95c57a45e5c1" exitCode=0
Jul 06 18:55:21.068844 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.067587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"4ffc68879924cf7df306a59374b206447037614e22bd7f4b37ba95c57a45e5c1"}
Jul 06 18:55:21.074440 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"cf1ee93d40bbbedfc4edd1a156cd698383e39efe2d7bd1a3fd755b47c01f66a9"}
Jul 06 18:55:21.074593 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074502    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"51593c7eceb5c87be82dd988d16273aaeda7864888980a76b3422b998c937016"}
Jul 06 18:55:21.074593 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"444957551beccdedf0e2d289a770aabe324f78337bd7d3a7702f9ff9e38c4fb8"}
Jul 06 18:55:21.074706 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"9b7d968cb9d17d70888b2c2792e8de94378b521f578936d0efa10488902aeaaf"}
Jul 06 18:55:21.074706 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074639    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"aca50ed43ebec01634f14364a384404bc47773a3522a2dffd9fef251b99eb2bc"}
Jul 06 18:55:21.074806 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.074772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"0c28686e0c31aec00e48f651989a751084821928470ccdaf74e243fd3c6fba54"}
Jul 06 18:55:21.076846 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.076822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j4jpp" event={"ID":"fcc43497-d4f3-4732-b4ab-a3c351f6b64e","Type":"ContainerStarted","Data":"d7e1099b0b884e5c4d1844a06309279e86f863d31792f98ea3a5128a9454510b"}
Jul 06 18:55:21.078560 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.078530    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk" event={"ID":"49d444ff-acd6-4036-8914-1cd3cbfe8edd","Type":"ContainerStarted","Data":"e3afa41e42b641df0c41388455fbfd9f653dd49a79ddeed0a01b9fc5939a6af5"}
Jul 06 18:55:21.080128 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.080095    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sh27w" event={"ID":"fe78c48b-7177-4cc0-908d-fbe69ed06e12","Type":"ContainerStarted","Data":"0e5649dd5d4807fe2f857c172b304ac8dec4d2cf8f94795054d4f1bbbed05dfb"}
Jul 06 18:55:21.081571 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.081549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q95qr" event={"ID":"cc8e9d9f-43be-4e26-b5f6-04e8177a4d47","Type":"ContainerStarted","Data":"7b52cfcd555fbb133c921949648c977bd8ca6de7cd616e606c222fa32a50a255"}
Jul 06 18:55:21.083362 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.083336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ml7kg" event={"ID":"ea88e9c0-af09-433a-a05b-bafaf2faacaf","Type":"ContainerStarted","Data":"970e9a0f9679ccb57349dbcf91d7207723054e9fa10850cf8dc46e0808233f5b"}
Jul 06 18:55:21.084722 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.084694    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zch6z" event={"ID":"8485ac65-9021-4ac1-97fb-cdc1bc397845","Type":"ContainerStarted","Data":"dc6093af8f78881ae71c884ba38316205f43238d0cff0e0abfa3d0aa12ee223c"}
Jul 06 18:55:21.098478 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.098410    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-123.ec2.internal" podStartSLOduration=22.098391547 podStartE2EDuration="22.098391547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:03.047696446 +0000 UTC m=+5.633904423" watchObservedRunningTime="2026-07-06 18:55:21.098391547 +0000 UTC m=+23.684599522"
Jul 06 18:55:21.128711 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.128658    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ml7kg" podStartSLOduration=3.8893918149999998 podStartE2EDuration="23.128642566s" totalImagesPullingTime="19.239250751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:21.124262728 +0000 UTC m=+23.710470703" watchObservedRunningTime="2026-07-06 18:55:21.128642566 +0000 UTC m=+23.714850539"
Jul 06 18:55:21.144956 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.144900    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zch6z" podStartSLOduration=8.592279524 podStartE2EDuration="23.144881305s" totalImagesPullingTime="14.552601781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:21.14426329 +0000 UTC m=+23.730471264" watchObservedRunningTime="2026-07-06 18:55:21.144881305 +0000 UTC m=+23.731089278"
Jul 06 18:55:21.168042 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.167992    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q95qr" podStartSLOduration=3.989075007 podStartE2EDuration="23.167976109s" totalImagesPullingTime="19.178901102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:21.167884605 +0000 UTC m=+23.754092579" watchObservedRunningTime="2026-07-06 18:55:21.167976109 +0000 UTC m=+23.754184083"
Jul 06 18:55:21.194168 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.194123    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sh27w" podStartSLOduration=3.981984896 podStartE2EDuration="23.194108192s" totalImagesPullingTime="19.212123296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:21.194010881 +0000 UTC m=+23.780218885" watchObservedRunningTime="2026-07-06 18:55:21.194108192 +0000 UTC m=+23.780316165"
Jul 06 18:55:21.224863 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.224819    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j4jpp" podStartSLOduration=4.055910417 podStartE2EDuration="23.224803835s" totalImagesPullingTime="19.168893418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:21.224697425 +0000 UTC m=+23.810905399" watchObservedRunningTime="2026-07-06 18:55:21.224803835 +0000 UTC m=+23.811011809"
Jul 06 18:55:21.482407 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.482385    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 18:55:21.918071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.917891    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T18:55:21.48240507Z","UUID":"e4ea2713-f73f-4a72-86cd-cae56b3c7f33","Handler":null,"Name":"","Endpoint":""}
Jul 06 18:55:21.919996 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.919963    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 18:55:21.920146 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.920013    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 18:55:21.956733 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.956703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:21.956919 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.956707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:21.956919 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:21.956824    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:21.957004 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:21.956908    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:21.957004 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:21.956707    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:21.957115 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:21.957023    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:22.089991 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:22.089950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk" event={"ID":"49d444ff-acd6-4036-8914-1cd3cbfe8edd","Type":"ContainerStarted","Data":"2491164f6bb89218550c614c0dd29db65f933ce1667d5ef914612b1669a727f5"}
Jul 06 18:55:22.093018 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:22.092989    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7jrxx" event={"ID":"5771f050-65b3-425d-9807-049c2b94d769","Type":"ContainerStarted","Data":"9a3e5c89c85cdf2fc88047fa0a559d4a991f559f2a8b590e3553e517b6b2552f"}
Jul 06 18:55:22.108906 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:22.108846    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7jrxx" podStartSLOduration=4.928702875 podStartE2EDuration="24.108824083s" totalImagesPullingTime="19.180121208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:22.108234619 +0000 UTC m=+24.694442594" watchObservedRunningTime="2026-07-06 18:55:22.108824083 +0000 UTC m=+24.695032058"
Jul 06 18:55:22.778617 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:22.778323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:22.779048 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:22.779027    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:23.098455 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.098414    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"5562cfe3526934a6bc6c71fe71602e5171abd0e622d995e54b63bcd286ef295e"}
Jul 06 18:55:23.100671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.100644    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk" event={"ID":"49d444ff-acd6-4036-8914-1cd3cbfe8edd","Type":"ContainerStarted","Data":"b571a4292f5e61a498494cf163e058ec4591f35723d8fa729e7d93825d4ea9b9"}
Jul 06 18:55:23.101182 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.101160    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:23.101659 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.101638    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zch6z"
Jul 06 18:55:23.123313 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.123259    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rblrk" podStartSLOduration=3.769293871 podStartE2EDuration="25.123237637s" totalImagesPullingTime="21.353943766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:23.122730526 +0000 UTC m=+25.708938504" watchObservedRunningTime="2026-07-06 18:55:23.123237637 +0000 UTC m=+25.709445614"
Jul 06 18:55:23.956417 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.956372    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:23.956652 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.956385    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:23.956652 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:23.956516    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:23.956652 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:23.956573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:23.956846 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:23.956656    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:23.956846 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:23.956734    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:25.956872 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:25.956619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:25.957651 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:25.956619    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:25.957651 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:25.956917    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:25.957651 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:25.956624    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:25.957651 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:25.956972    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:25.957651 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:25.957049    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:26.110286 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.110251    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="fcba7aae9f1fa92c5c021c9f661d6dcc4333e568dabcc8d0bc359dec85e10de9" exitCode=0
Jul 06 18:55:26.110483 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.110329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"fcba7aae9f1fa92c5c021c9f661d6dcc4333e568dabcc8d0bc359dec85e10de9"}
Jul 06 18:55:26.114166 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.114143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" event={"ID":"d8c36617-d4dc-47f9-a660-607fff2508ab","Type":"ContainerStarted","Data":"291ddefbc3974ad48ecb44d83b6e7f73ce8037486483c48fb6a225f2313bd149"}
Jul 06 18:55:26.114450 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.114428    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:26.132324 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.132297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:26.169504 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:26.169441    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm" podStartSLOduration=8.872329676 podStartE2EDuration="28.169425014s" totalImagesPullingTime="19.297095338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:26.168939971 +0000 UTC m=+28.755147946" watchObservedRunningTime="2026-07-06 18:55:26.169425014 +0000 UTC m=+28.755632988"
Jul 06 18:55:27.117743 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.117715    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:27.118247 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.117791    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:27.134253 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.134226    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:27.607021 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.606811    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56wjc"]
Jul 06 18:55:27.607183 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.607113    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:27.607224 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:27.607209    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:27.610110 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.610082    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2w7v7"]
Jul 06 18:55:27.610261 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.610211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:27.610346 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:27.610324    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:27.610800 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.610780    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6f5c8"]
Jul 06 18:55:27.610898 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:27.610885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:27.611012 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:27.610992    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:28.121440 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:28.121400    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="cf36fadcfb21bd1f17b9cb08cf36a1898002bced32ab3f05b21aa7fc65289c0c" exitCode=0
Jul 06 18:55:28.121964 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:28.121495    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"cf36fadcfb21bd1f17b9cb08cf36a1898002bced32ab3f05b21aa7fc65289c0c"}
Jul 06 18:55:28.956910 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:28.956876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:28.957060 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:28.956996    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:29.956754 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:29.956661    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:29.957424 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:29.956788    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:29.957424 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:29.956851    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:29.957424 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:29.956959    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:30.128568 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:30.128532    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="0c9b256abc9b05cc56aca31dae6d192b81f93e65975b9ed43bb3a5f9ea946bb1" exitCode=0
Jul 06 18:55:30.128718 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:30.128577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"0c9b256abc9b05cc56aca31dae6d192b81f93e65975b9ed43bb3a5f9ea946bb1"}
Jul 06 18:55:30.956116 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:30.956081    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:30.956309 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:30.956197    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2w7v7" podUID="7c2f4853-e8c2-47f1-9ef4-f41e73e2e716"
Jul 06 18:55:31.598220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:31.598176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:31.598687 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.598379    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:31.598687 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.598480    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs podName:b28c3344-93f0-4722-b2dd-c3a0e258839f nodeName:}" failed. No retries permitted until 2026-07-06 18:56:03.598440417 +0000 UTC m=+66.184648372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs") pod "network-metrics-daemon-6f5c8" (UID: "b28c3344-93f0-4722-b2dd-c3a0e258839f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 18:55:31.698895 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:31.698860    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:31.699087 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.699054    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:31.699173 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.699149    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret podName:093150fe-2f19-46de-a8e3-714e7ee0779c nodeName:}" failed. No retries permitted until 2026-07-06 18:56:03.699128796 +0000 UTC m=+66.285336763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret") pod "global-pull-secret-syncer-56wjc" (UID: "093150fe-2f19-46de-a8e3-714e7ee0779c") : object "kube-system"/"original-pull-secret" not registered
Jul 06 18:55:31.799653 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:31.799568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:31.799808 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.799786    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 18:55:31.799867 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.799814    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 18:55:31.799867 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.799827    2561 projected.go:196] Error preparing data for projected volume kube-api-access-stn8k for pod openshift-network-diagnostics/network-check-target-2w7v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:31.799955 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.799897    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k podName:7c2f4853-e8c2-47f1-9ef4-f41e73e2e716 nodeName:}" failed. No retries permitted until 2026-07-06 18:56:03.799876535 +0000 UTC m=+66.386084492 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-stn8k" (UniqueName: "kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k") pod "network-check-target-2w7v7" (UID: "7c2f4853-e8c2-47f1-9ef4-f41e73e2e716") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 18:55:31.956583 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:31.956547    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:31.956763 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.956682    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6f5c8" podUID="b28c3344-93f0-4722-b2dd-c3a0e258839f"
Jul 06 18:55:31.956763 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:31.956736    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:31.956871 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:31.956848    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-56wjc" podUID="093150fe-2f19-46de-a8e3-714e7ee0779c"
Jul 06 18:55:32.624697 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.624482    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-123.ec2.internal" event="NodeReady"
Jul 06 18:55:32.625106 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.624834    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 18:55:32.677229 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.677196    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-596vd"]
Jul 06 18:55:32.681235 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.681210    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:55:32.681377 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.681339    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.684399 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.684161    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 18:55:32.684399 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.684190    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q4rjh\""
Jul 06 18:55:32.684399 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.684350    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 18:55:32.685084 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.685066    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ln5nv"]
Jul 06 18:55:32.685209 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.685191    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.687561 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.687537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 18:55:32.687689 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.687558    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 18:55:32.687689 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.687619    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 18:55:32.688084 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.688069    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48zcw\""
Jul 06 18:55:32.691978 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.691951    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-596vd"]
Jul 06 18:55:32.692186 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.692168    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.699272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.696170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 18:55:32.699272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.696757    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 18:55:32.699272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.697669    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ft894\""
Jul 06 18:55:32.699272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.698875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ln5nv"]
Jul 06 18:55:32.699272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.699032    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 18:55:32.701061 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.700976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 18:55:32.701209 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.701185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:55:32.807758 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqf7t\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807784    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea86a2af-8ed1-43ea-b256-636f7563fb0c-tmp-dir\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea86a2af-8ed1-43ea-b256-636f7563fb0c-config-volume\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.807913 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807897    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rl47\" (UniqueName: \"kubernetes.io/projected/ea86a2af-8ed1-43ea-b256-636f7563fb0c-kube-api-access-7rl47\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.807982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.808040    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxqz\" (UniqueName: \"kubernetes.io/projected/3c3e2b89-9407-4bed-b547-c2a6b1681311-kube-api-access-4sxqz\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.808058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.808079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.808097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.808155 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.808115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3e2b89-9407-4bed-b547-c2a6b1681311-cert\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.909316 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea86a2af-8ed1-43ea-b256-636f7563fb0c-tmp-dir\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.909316 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.909633 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909609    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea86a2af-8ed1-43ea-b256-636f7563fb0c-tmp-dir\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.909730 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909678    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea86a2af-8ed1-43ea-b256-636f7563fb0c-config-volume\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.909730 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.909832 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.909832 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.909922 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:32.909848    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 18:55:32.909922 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909865    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.909922 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rl47\" (UniqueName: \"kubernetes.io/projected/ea86a2af-8ed1-43ea-b256-636f7563fb0c-kube-api-access-7rl47\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.909922 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:32.909908    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls podName:ea86a2af-8ed1-43ea-b256-636f7563fb0c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:33.409889584 +0000 UTC m=+35.996097556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls") pod "dns-default-596vd" (UID: "ea86a2af-8ed1-43ea-b256-636f7563fb0c") : secret "dns-default-metrics-tls" not found
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909926    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909965    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxqz\" (UniqueName: \"kubernetes.io/projected/3c3e2b89-9407-4bed-b547-c2a6b1681311-kube-api-access-4sxqz\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.909992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910057    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910059    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910122 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3e2b89-9407-4bed-b547-c2a6b1681311-cert\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.910426 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqf7t\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.910426 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.910310    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea86a2af-8ed1-43ea-b256-636f7563fb0c-config-volume\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.910426 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:32.910397    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 18:55:32.910426 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:32.910410    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bffd688bb-l594z: secret "image-registry-tls" not found
Jul 06 18:55:32.910626 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:32.910452    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls podName:014d0aa6-8fbd-461f-8dd7-48bad028504b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:33.410438258 +0000 UTC m=+35.996646227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls") pod "image-registry-6bffd688bb-l594z" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b") : secret "image-registry-tls" not found
Jul 06 18:55:32.911916 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.911765    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.915167 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.915144    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.915283 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.915210    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c3e2b89-9407-4bed-b547-c2a6b1681311-cert\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.915524 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.915507    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.919880 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.919853    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.920076 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.920055    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rl47\" (UniqueName: \"kubernetes.io/projected/ea86a2af-8ed1-43ea-b256-636f7563fb0c-kube-api-access-7rl47\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:32.920175 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.920157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxqz\" (UniqueName: \"kubernetes.io/projected/3c3e2b89-9407-4bed-b547-c2a6b1681311-kube-api-access-4sxqz\") pod \"ingress-canary-ln5nv\" (UID: \"3c3e2b89-9407-4bed-b547-c2a6b1681311\") " pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:32.921019 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.920995    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqf7t\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:32.956725 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.956685    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:55:32.960107 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.960081    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 18:55:32.960245 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.960116    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk8fr\""
Jul 06 18:55:32.960371 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:32.960355    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 18:55:33.014971 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.014929    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ln5nv"
Jul 06 18:55:33.413938 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.413903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:33.414124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.413989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:33.414124 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:33.414081    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 18:55:33.414124 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:33.414094    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 18:55:33.414124 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:33.414106    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bffd688bb-l594z: secret "image-registry-tls" not found
Jul 06 18:55:33.414314 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:33.414160    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls podName:ea86a2af-8ed1-43ea-b256-636f7563fb0c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:34.414135183 +0000 UTC m=+37.000343138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls") pod "dns-default-596vd" (UID: "ea86a2af-8ed1-43ea-b256-636f7563fb0c") : secret "dns-default-metrics-tls" not found
Jul 06 18:55:33.414314 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:33.414182    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls podName:014d0aa6-8fbd-461f-8dd7-48bad028504b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:34.414170962 +0000 UTC m=+37.000378919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls") pod "image-registry-6bffd688bb-l594z" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b") : secret "image-registry-tls" not found
Jul 06 18:55:33.956166 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.956123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:55:33.956633 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.956141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:55:33.959347 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.959318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 18:55:33.960428 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.960400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w7zz5\""
Jul 06 18:55:33.960579 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:33.960426    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 18:55:34.424560 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:34.424529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:34.424701 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:34.424598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:34.424701 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:34.424649    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 18:55:34.424813 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:34.424722    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls podName:ea86a2af-8ed1-43ea-b256-636f7563fb0c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:36.424699573 +0000 UTC m=+39.010907527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls") pod "dns-default-596vd" (UID: "ea86a2af-8ed1-43ea-b256-636f7563fb0c") : secret "dns-default-metrics-tls" not found
Jul 06 18:55:34.424813 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:34.424741    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 18:55:34.424813 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:34.424755    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bffd688bb-l594z: secret "image-registry-tls" not found
Jul 06 18:55:34.424813 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:34.424806    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls podName:014d0aa6-8fbd-461f-8dd7-48bad028504b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:36.424794201 +0000 UTC m=+39.011002155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls") pod "image-registry-6bffd688bb-l594z" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b") : secret "image-registry-tls" not found
Jul 06 18:55:34.508183 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:34.508153    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ln5nv"]
Jul 06 18:55:34.511553 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:34.511527    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3e2b89_9407_4bed_b547_c2a6b1681311.slice/crio-e96be8ffc7127d86e867d39a73784a7eaea250892a344ba17d3c19dec443cbff WatchSource:0}: Error finding container e96be8ffc7127d86e867d39a73784a7eaea250892a344ba17d3c19dec443cbff: Status 404 returned error can't find the container with id e96be8ffc7127d86e867d39a73784a7eaea250892a344ba17d3c19dec443cbff
Jul 06 18:55:35.144053 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:35.144008    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="9ff63cc5d6b236fc076249689451a00e2ace100735f2f4c608fdb918e763374b" exitCode=0
Jul 06 18:55:35.144483 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:35.144097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"9ff63cc5d6b236fc076249689451a00e2ace100735f2f4c608fdb918e763374b"}
Jul 06 18:55:35.145507 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:35.145476    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ln5nv" event={"ID":"3c3e2b89-9407-4bed-b547-c2a6b1681311","Type":"ContainerStarted","Data":"e96be8ffc7127d86e867d39a73784a7eaea250892a344ba17d3c19dec443cbff"}
Jul 06 18:55:36.152369 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.152330    2561 generic.go:356] "Generic (PLEG): container finished" podID="52bea5b7-0315-4919-beb2-4954ccfea43a" containerID="3bd1a7010dd5c479a478530bc390a97ae68248e5901f62cbf141e6401f6841d0" exitCode=0
Jul 06 18:55:36.152920 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.152394    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerDied","Data":"3bd1a7010dd5c479a478530bc390a97ae68248e5901f62cbf141e6401f6841d0"}
Jul 06 18:55:36.260894 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.260858    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"]
Jul 06 18:55:36.265788 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.265762    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:36.269925 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.269891    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 06 18:55:36.270084 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.270050    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h7w9t\""
Jul 06 18:55:36.270355 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.270335    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 06 18:55:36.273068 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.273038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"]
Jul 06 18:55:36.442419 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.442326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6fccd601-0088-4800-864d-0c1b7bf65c64-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:36.442419 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.442396    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:36.442670 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.442521    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 18:55:36.442670 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.442529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:36.442670 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.442577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:36.442670 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.442595    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls podName:ea86a2af-8ed1-43ea-b256-636f7563fb0c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:40.442577274 +0000 UTC m=+43.028785228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls") pod "dns-default-596vd" (UID: "ea86a2af-8ed1-43ea-b256-636f7563fb0c") : secret "dns-default-metrics-tls" not found
Jul 06 18:55:36.442865 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.442681    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 18:55:36.442865 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.442696    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bffd688bb-l594z: secret "image-registry-tls" not found
Jul 06 18:55:36.442865 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.442767    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls podName:014d0aa6-8fbd-461f-8dd7-48bad028504b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:40.44274924 +0000 UTC m=+43.028957199 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls") pod "image-registry-6bffd688bb-l594z" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b") : secret "image-registry-tls" not found
Jul 06 18:55:36.544082 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.544043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:36.544255 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.544111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6fccd601-0088-4800-864d-0c1b7bf65c64-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:36.544255 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.544217    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:36.544373 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:36.544300    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:37.044283642 +0000 UTC m=+39.630491593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:36.544866 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:36.544844    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6fccd601-0088-4800-864d-0c1b7bf65c64-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:37.048486 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:37.048430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:37.048645 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:37.048588    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:37.048681 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:37.048653    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:38.048636763 +0000 UTC m=+40.634844715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:37.158180 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:37.158097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" event={"ID":"52bea5b7-0315-4919-beb2-4954ccfea43a","Type":"ContainerStarted","Data":"838fc8660dca3c7a1ae38539cd2c49a7db532953a9bdc54718f729242d217482"}
Jul 06 18:55:37.189486 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:37.189420    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dlp6k" podStartSLOduration=9.091371219 podStartE2EDuration="39.189404577s" totalImagesPullingTime="30.098033358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:37.188149635 +0000 UTC m=+39.774357608" watchObservedRunningTime="2026-07-06 18:55:37.189404577 +0000 UTC m=+39.775612553"
Jul 06 18:55:38.002220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.002185    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pz46j"]
Jul 06 18:55:38.027942 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.027905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pz46j"]
Jul 06 18:55:38.028123 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.028038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.033006 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.032977    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 18:55:38.033252 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.033231    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 18:55:38.033426 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.033406    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 18:55:38.033555 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.033310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 18:55:38.037816 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.037797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tlz5r\""
Jul 06 18:55:38.054511 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.054451    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:38.054689 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.054611    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:38.054729 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.054692    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:40.054675091 +0000 UTC m=+42.640883046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:38.155657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.155618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.155834 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.155671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pt2c\" (UniqueName: \"kubernetes.io/projected/429a4284-f11f-4db9-a202-4e6691a3961b-kube-api-access-8pt2c\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.155834 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.155743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.155834 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.155797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/429a4284-f11f-4db9-a202-4e6691a3961b-data-volume\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.155933 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.155845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/429a4284-f11f-4db9-a202-4e6691a3961b-crio-socket\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.161992 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.161962    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ln5nv" event={"ID":"3c3e2b89-9407-4bed-b547-c2a6b1681311","Type":"ContainerStarted","Data":"6d478eca0d763eeff615ea9dba9700348a5163fd5ff8b23bb8b4e7fc4d0ec582"}
Jul 06 18:55:38.190429 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.190383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ln5nv" podStartSLOduration=3.46050873 podStartE2EDuration="6.190368726s" totalImagesPullingTime="2.729859996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:38.190164683 +0000 UTC m=+40.776372676" watchObservedRunningTime="2026-07-06 18:55:38.190368726 +0000 UTC m=+40.776576700"
Jul 06 18:55:38.199998 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.199964    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"]
Jul 06 18:55:38.221826 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.221799    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"]
Jul 06 18:55:38.222058 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.222031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"
Jul 06 18:55:38.224819 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.224797    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xw4lf\""
Jul 06 18:55:38.224996 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.224892    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 06 18:55:38.224996 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.224966    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 06 18:55:38.257182 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257085    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.257355 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/429a4284-f11f-4db9-a202-4e6691a3961b-data-volume\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.257355 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/429a4284-f11f-4db9-a202-4e6691a3961b-crio-socket\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.257355 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.257214    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:38.257511 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.257383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:38.757363726 +0000 UTC m=+41.343571681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:38.257822 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257787    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/429a4284-f11f-4db9-a202-4e6691a3961b-crio-socket\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.257956 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257883    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/429a4284-f11f-4db9-a202-4e6691a3961b-data-volume\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.257956 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.257886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.258060 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.257953    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 18:55:38.258060 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.258016    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:38.757996878 +0000 UTC m=+41.344204843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : configmap "kube-rbac-proxy" not found
Jul 06 18:55:38.258158 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.258141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pt2c\" (UniqueName: \"kubernetes.io/projected/429a4284-f11f-4db9-a202-4e6691a3961b-kube-api-access-8pt2c\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.269737 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.269714    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pt2c\" (UniqueName: \"kubernetes.io/projected/429a4284-f11f-4db9-a202-4e6691a3961b-kube-api-access-8pt2c\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.358711 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.358672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdbq\" (UniqueName: \"kubernetes.io/projected/be3fbda2-88be-41df-ac72-f75e40e5f29e-kube-api-access-mzdbq\") pod \"migrator-d57d58cc9-bxdql\" (UID: \"be3fbda2-88be-41df-ac72-f75e40e5f29e\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"
Jul 06 18:55:38.459550 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.459510    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdbq\" (UniqueName: \"kubernetes.io/projected/be3fbda2-88be-41df-ac72-f75e40e5f29e-kube-api-access-mzdbq\") pod \"migrator-d57d58cc9-bxdql\" (UID: \"be3fbda2-88be-41df-ac72-f75e40e5f29e\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"
Jul 06 18:55:38.468922 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.468893    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdbq\" (UniqueName: \"kubernetes.io/projected/be3fbda2-88be-41df-ac72-f75e40e5f29e-kube-api-access-mzdbq\") pod \"migrator-d57d58cc9-bxdql\" (UID: \"be3fbda2-88be-41df-ac72-f75e40e5f29e\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"
Jul 06 18:55:38.531134 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.531032    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"
Jul 06 18:55:38.676944 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.676909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql"]
Jul 06 18:55:38.680087 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:38.680056    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3fbda2_88be_41df_ac72_f75e40e5f29e.slice/crio-3b9b631b7ceda3070b6370e3381190d212119be60ab1cde12390c7687fa3ec90 WatchSource:0}: Error finding container 3b9b631b7ceda3070b6370e3381190d212119be60ab1cde12390c7687fa3ec90: Status 404 returned error can't find the container with id 3b9b631b7ceda3070b6370e3381190d212119be60ab1cde12390c7687fa3ec90
Jul 06 18:55:38.762272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.762239    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.762447 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:38.762296    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:38.762447 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.762350    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 18:55:38.762447 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.762395    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:38.762447 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.762414    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:39.762397472 +0000 UTC m=+42.348605423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : configmap "kube-rbac-proxy" not found
Jul 06 18:55:38.762447 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:38.762445    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:39.762431745 +0000 UTC m=+42.348639696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:39.166220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:39.166181    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql" event={"ID":"be3fbda2-88be-41df-ac72-f75e40e5f29e","Type":"ContainerStarted","Data":"3b9b631b7ceda3070b6370e3381190d212119be60ab1cde12390c7687fa3ec90"}
Jul 06 18:55:39.770972 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:39.770937    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:39.771160 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:39.770995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:39.771160 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:39.771072    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 18:55:39.771160 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:39.771103    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:39.771160 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:39.771162    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:41.771147631 +0000 UTC m=+44.357355583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:39.771326 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:39.771176    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:41.771169957 +0000 UTC m=+44.357377910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : configmap "kube-rbac-proxy" not found
Jul 06 18:55:39.800576 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:39.800546    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q95qr_cc8e9d9f-43be-4e26-b5f6-04e8177a4d47/dns-node-resolver/0.log"
Jul 06 18:55:40.074198 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.074174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:40.074321 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.074304    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:40.074374 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.074365    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:44.074350923 +0000 UTC m=+46.660558875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:40.401275 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.401202    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j4jpp_fcc43497-d4f3-4732-b4ab-a3c351f6b64e/node-ca/0.log"
Jul 06 18:55:40.432936 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.432898    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5zfll"]
Jul 06 18:55:40.456534 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.456501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5zfll"]
Jul 06 18:55:40.456671 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.456573    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.460212 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.460189    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 06 18:55:40.460352 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.460333    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 06 18:55:40.461106 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.461083    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-szlnj\""
Jul 06 18:55:40.461209 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.461116    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 06 18:55:40.461209 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.461082    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 06 18:55:40.477493 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.477442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:40.477660 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.477556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:40.477660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.477574    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 18:55:40.477660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.477593    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6bffd688bb-l594z: secret "image-registry-tls" not found
Jul 06 18:55:40.477660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.477652    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls podName:014d0aa6-8fbd-461f-8dd7-48bad028504b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:48.477634661 +0000 UTC m=+51.063842629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls") pod "image-registry-6bffd688bb-l594z" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b") : secret "image-registry-tls" not found
Jul 06 18:55:40.477660 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.477655    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 18:55:40.477821 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:40.477700    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls podName:ea86a2af-8ed1-43ea-b256-636f7563fb0c nodeName:}" failed. No retries permitted until 2026-07-06 18:55:48.477684033 +0000 UTC m=+51.063891986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls") pod "dns-default-596vd" (UID: "ea86a2af-8ed1-43ea-b256-636f7563fb0c") : secret "dns-default-metrics-tls" not found
Jul 06 18:55:40.578357 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.578316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxzfk\" (UniqueName: \"kubernetes.io/projected/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-kube-api-access-dxzfk\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.578582 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.578381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-cabundle\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.578582 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.578410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-key\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.679714 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.679623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-cabundle\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.679714 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.679676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-key\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.679890 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.679806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxzfk\" (UniqueName: \"kubernetes.io/projected/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-kube-api-access-dxzfk\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.680398 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.680380    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-cabundle\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.682201 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.682180    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-signing-key\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.690274 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.690248    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxzfk\" (UniqueName: \"kubernetes.io/projected/1d0c4584-4bbb-4d8e-815b-9091d81bbad4-kube-api-access-dxzfk\") pod \"service-ca-758f599c5-5zfll\" (UID: \"1d0c4584-4bbb-4d8e-815b-9091d81bbad4\") " pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.765554 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.765511    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5zfll"
Jul 06 18:55:40.936813 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:40.936730    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5zfll"]
Jul 06 18:55:40.939738 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:40.939711    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0c4584_4bbb_4d8e_815b_9091d81bbad4.slice/crio-2269bb354e3a0fda78978e9b15162b8b91811afba307f9a69ae380f5c3667006 WatchSource:0}: Error finding container 2269bb354e3a0fda78978e9b15162b8b91811afba307f9a69ae380f5c3667006: Status 404 returned error can't find the container with id 2269bb354e3a0fda78978e9b15162b8b91811afba307f9a69ae380f5c3667006
Jul 06 18:55:41.172583 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.172543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql" event={"ID":"be3fbda2-88be-41df-ac72-f75e40e5f29e","Type":"ContainerStarted","Data":"ac81ccffd4529b5c65501278431e2230f684eca86f666ed2d3bf2c068baf66f1"}
Jul 06 18:55:41.172780 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.172603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql" event={"ID":"be3fbda2-88be-41df-ac72-f75e40e5f29e","Type":"ContainerStarted","Data":"01cb18babb4ce1d1c74b868152be24c13e1cea5c4f98a45dd0dcff42890eade4"}
Jul 06 18:55:41.173637 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.173613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5zfll" event={"ID":"1d0c4584-4bbb-4d8e-815b-9091d81bbad4","Type":"ContainerStarted","Data":"2269bb354e3a0fda78978e9b15162b8b91811afba307f9a69ae380f5c3667006"}
Jul 06 18:55:41.193327 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.193227    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-bxdql" podStartSLOduration=1.800077326 podStartE2EDuration="3.19321179s" totalImagesPullingTime="1.393134464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:41.192595088 +0000 UTC m=+43.778803061" watchObservedRunningTime="2026-07-06 18:55:41.19321179 +0000 UTC m=+43.779419797"
Jul 06 18:55:41.199435 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.199396    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ln5nv_3c3e2b89-9407-4bed-b547-c2a6b1681311/serve-healthcheck-canary/0.log"
Jul 06 18:55:41.789797 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.789758    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:41.790291 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:41.789851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:41.790291 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:41.789953    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 18:55:41.790291 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:41.790053    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:45.79003081 +0000 UTC m=+48.376238776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : configmap "kube-rbac-proxy" not found
Jul 06 18:55:41.790291 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:41.790101    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:41.790291 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:41.790180    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:45.790162655 +0000 UTC m=+48.376370624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:43.179815 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:43.179784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5zfll" event={"ID":"1d0c4584-4bbb-4d8e-815b-9091d81bbad4","Type":"ContainerStarted","Data":"26d6a362fb6d6faf1bb676c4f85b9e3475048c1d35ad6d346562064e6c41620e"}
Jul 06 18:55:43.202384 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:43.202319    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5zfll" podStartSLOduration=1.134201149 podStartE2EDuration="3.202295815s" totalImagesPullingTime="2.068094666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:43.200515397 +0000 UTC m=+45.786723368" watchObservedRunningTime="2026-07-06 18:55:43.202295815 +0000 UTC m=+45.788503790"
Jul 06 18:55:44.111806 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:44.111763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:44.112005 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:44.111891    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:44.112005 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:44.111947    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:55:52.111932983 +0000 UTC m=+54.698140935 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:45.826542 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:45.826505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:45.826974 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:45.826570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:45.826974 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:45.826731    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:45.826974 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:45.826809    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls podName:429a4284-f11f-4db9-a202-4e6691a3961b nodeName:}" failed. No retries permitted until 2026-07-06 18:55:53.826793035 +0000 UTC m=+56.413000990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pz46j" (UID: "429a4284-f11f-4db9-a202-4e6691a3961b") : secret "insights-runtime-extractor-tls" not found
Jul 06 18:55:45.827097 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:45.827051    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/429a4284-f11f-4db9-a202-4e6691a3961b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:48.549515 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.549454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:48.549902 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.549552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:48.552021 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.551992    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea86a2af-8ed1-43ea-b256-636f7563fb0c-metrics-tls\") pod \"dns-default-596vd\" (UID: \"ea86a2af-8ed1-43ea-b256-636f7563fb0c\") " pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:48.552143 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.552030    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"image-registry-6bffd688bb-l594z\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") " pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:48.600183 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.600155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:48.609317 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.609285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:48.759713 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.759677    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-596vd"]
Jul 06 18:55:48.763289 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:48.763255    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea86a2af_8ed1_43ea_b256_636f7563fb0c.slice/crio-5b48d295776556d043ecccbfa065e337aed0f50c45af50ed69677d53ff37d18a WatchSource:0}: Error finding container 5b48d295776556d043ecccbfa065e337aed0f50c45af50ed69677d53ff37d18a: Status 404 returned error can't find the container with id 5b48d295776556d043ecccbfa065e337aed0f50c45af50ed69677d53ff37d18a
Jul 06 18:55:48.778903 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:48.778874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:55:48.782175 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:48.782135    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014d0aa6_8fbd_461f_8dd7_48bad028504b.slice/crio-fe1735800309280c14c43faf24537fc9c5e9dc5949a3c7189095ec97638b8b1e WatchSource:0}: Error finding container fe1735800309280c14c43faf24537fc9c5e9dc5949a3c7189095ec97638b8b1e: Status 404 returned error can't find the container with id fe1735800309280c14c43faf24537fc9c5e9dc5949a3c7189095ec97638b8b1e
Jul 06 18:55:49.198140 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:49.198032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" event={"ID":"014d0aa6-8fbd-461f-8dd7-48bad028504b","Type":"ContainerStarted","Data":"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"}
Jul 06 18:55:49.198140 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:49.198093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" event={"ID":"014d0aa6-8fbd-461f-8dd7-48bad028504b","Type":"ContainerStarted","Data":"fe1735800309280c14c43faf24537fc9c5e9dc5949a3c7189095ec97638b8b1e"}
Jul 06 18:55:49.198140 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:49.198120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:55:49.199273 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:49.199244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-596vd" event={"ID":"ea86a2af-8ed1-43ea-b256-636f7563fb0c","Type":"ContainerStarted","Data":"5b48d295776556d043ecccbfa065e337aed0f50c45af50ed69677d53ff37d18a"}
Jul 06 18:55:49.218535 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:49.218476    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" podStartSLOduration=32.218436242 podStartE2EDuration="32.218436242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:49.217857632 +0000 UTC m=+51.804065619" watchObservedRunningTime="2026-07-06 18:55:49.218436242 +0000 UTC m=+51.804644218"
Jul 06 18:55:52.181011 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:52.180976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:55:52.181354 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:52.181141    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 06 18:55:52.181354 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:55:52.181220    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert podName:6fccd601-0088-4800-864d-0c1b7bf65c64 nodeName:}" failed. No retries permitted until 2026-07-06 18:56:08.181199494 +0000 UTC m=+70.767407457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-5hnm4" (UID: "6fccd601-0088-4800-864d-0c1b7bf65c64") : secret "networking-console-plugin-cert" not found
Jul 06 18:55:53.213510 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.213449    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-596vd" event={"ID":"ea86a2af-8ed1-43ea-b256-636f7563fb0c","Type":"ContainerStarted","Data":"fa513d4083574bd866cd2b19c7c4f3b4e27343bcb9c91fb5f47331e5d08d9e94"}
Jul 06 18:55:53.213986 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.213526    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-596vd" event={"ID":"ea86a2af-8ed1-43ea-b256-636f7563fb0c","Type":"ContainerStarted","Data":"c386b1d4eabe1e418d37c1b4543afd1e8f7dff136de8697a58aba34da0a608d7"}
Jul 06 18:55:53.213986 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.213586    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-596vd"
Jul 06 18:55:53.234194 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.234131    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-596vd" podStartSLOduration=17.849983378 podStartE2EDuration="21.234108197s" totalImagesPullingTime="3.384124819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:55:53.23192552 +0000 UTC m=+55.818133496" watchObservedRunningTime="2026-07-06 18:55:53.234108197 +0000 UTC m=+55.820316173"
Jul 06 18:55:53.895121 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.895073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:53.897509 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.897479    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/429a4284-f11f-4db9-a202-4e6691a3961b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pz46j\" (UID: \"429a4284-f11f-4db9-a202-4e6691a3961b\") " pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:53.937571 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:53.937535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pz46j"
Jul 06 18:55:54.071013 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:54.070973    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pz46j"]
Jul 06 18:55:54.074412 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:55:54.074377    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429a4284_f11f_4db9_a202_4e6691a3961b.slice/crio-1bfe9c3187cdfc1b510805ba760d989cc15f5fb777868865a8074de3f950d3a1 WatchSource:0}: Error finding container 1bfe9c3187cdfc1b510805ba760d989cc15f5fb777868865a8074de3f950d3a1: Status 404 returned error can't find the container with id 1bfe9c3187cdfc1b510805ba760d989cc15f5fb777868865a8074de3f950d3a1
Jul 06 18:55:54.220698 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:54.220592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz46j" event={"ID":"429a4284-f11f-4db9-a202-4e6691a3961b","Type":"ContainerStarted","Data":"1bfe9c3187cdfc1b510805ba760d989cc15f5fb777868865a8074de3f950d3a1"}
Jul 06 18:55:59.140767 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:59.140733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fgzhm"
Jul 06 18:55:59.242143 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:55:59.242103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz46j" event={"ID":"429a4284-f11f-4db9-a202-4e6691a3961b","Type":"ContainerStarted","Data":"5c59df105d49508037f3ef690dcc84ce4646d82f137704e285023de3eba81d83"}
Jul 06 18:56:00.251073 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:00.251036    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz46j" event={"ID":"429a4284-f11f-4db9-a202-4e6691a3961b","Type":"ContainerStarted","Data":"ec23ebd3d6b9e72c4ce1363c5e8b7c57233f4f34dec786b64b2db82a0d6813b5"}
Jul 06 18:56:02.259737 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:02.259697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pz46j" event={"ID":"429a4284-f11f-4db9-a202-4e6691a3961b","Type":"ContainerStarted","Data":"c219755f4ff0d2f8c5221101a739cf90d07e36e3ed1cfe9d1e0bb7b114b3710f"}
Jul 06 18:56:02.278888 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:02.278827    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pz46j" podStartSLOduration=17.43342932 podStartE2EDuration="25.278811561s" totalImagesPullingTime="7.845382241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:55:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:02.277975635 +0000 UTC m=+64.864183614" watchObservedRunningTime="2026-07-06 18:56:02.278811561 +0000 UTC m=+64.865019535"
Jul 06 18:56:03.222982 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.222947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-596vd"
Jul 06 18:56:03.680152 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.680111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:56:03.682725 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.682707    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 18:56:03.693308 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.693276    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c3344-93f0-4722-b2dd-c3a0e258839f-metrics-certs\") pod \"network-metrics-daemon-6f5c8\" (UID: \"b28c3344-93f0-4722-b2dd-c3a0e258839f\") " pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:56:03.780924 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.780885    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:56:03.783859 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.783836    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 18:56:03.794197 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.794173    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/093150fe-2f19-46de-a8e3-714e7ee0779c-original-pull-secret\") pod \"global-pull-secret-syncer-56wjc\" (UID: \"093150fe-2f19-46de-a8e3-714e7ee0779c\") " pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:56:03.881928 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.881887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:56:03.884723 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.884705    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 18:56:03.895179 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.895158    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 18:56:03.905932 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.905895    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stn8k\" (UniqueName: \"kubernetes.io/projected/7c2f4853-e8c2-47f1-9ef4-f41e73e2e716-kube-api-access-stn8k\") pod \"network-check-target-2w7v7\" (UID: \"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716\") " pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:56:03.967733 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.967660    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-56wjc"
Jul 06 18:56:03.975456 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.975431    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w7zz5\""
Jul 06 18:56:03.982915 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:03.982892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6f5c8"
Jul 06 18:56:04.117694 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.117666    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-56wjc"]
Jul 06 18:56:04.121138 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:04.121095    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093150fe_2f19_46de_a8e3_714e7ee0779c.slice/crio-a6af7a901b4167dff6b9010dc3aa3a8a7b1e3452d52cc57a4738b8591be8c3ae WatchSource:0}: Error finding container a6af7a901b4167dff6b9010dc3aa3a8a7b1e3452d52cc57a4738b8591be8c3ae: Status 404 returned error can't find the container with id a6af7a901b4167dff6b9010dc3aa3a8a7b1e3452d52cc57a4738b8591be8c3ae
Jul 06 18:56:04.138297 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.138269    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6f5c8"]
Jul 06 18:56:04.141839 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:04.141815    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28c3344_93f0_4722_b2dd_c3a0e258839f.slice/crio-1c732f40ea5d4763f3beab9d05639f92b0b35fdaea94b5754a96b852335180d4 WatchSource:0}: Error finding container 1c732f40ea5d4763f3beab9d05639f92b0b35fdaea94b5754a96b852335180d4: Status 404 returned error can't find the container with id 1c732f40ea5d4763f3beab9d05639f92b0b35fdaea94b5754a96b852335180d4
Jul 06 18:56:04.169345 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.169320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gk8fr\""
Jul 06 18:56:04.177520 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.177495    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:56:04.268843 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.268807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-56wjc" event={"ID":"093150fe-2f19-46de-a8e3-714e7ee0779c","Type":"ContainerStarted","Data":"a6af7a901b4167dff6b9010dc3aa3a8a7b1e3452d52cc57a4738b8591be8c3ae"}
Jul 06 18:56:04.270162 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.270129    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6f5c8" event={"ID":"b28c3344-93f0-4722-b2dd-c3a0e258839f","Type":"ContainerStarted","Data":"1c732f40ea5d4763f3beab9d05639f92b0b35fdaea94b5754a96b852335180d4"}
Jul 06 18:56:04.316583 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:04.316553    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2w7v7"]
Jul 06 18:56:04.319961 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:04.319930    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2f4853_e8c2_47f1_9ef4_f41e73e2e716.slice/crio-2b6c3007af576d93be5563a04048987570cd2b3057f4efad3f85edb6295c5bef WatchSource:0}: Error finding container 2b6c3007af576d93be5563a04048987570cd2b3057f4efad3f85edb6295c5bef: Status 404 returned error can't find the container with id 2b6c3007af576d93be5563a04048987570cd2b3057f4efad3f85edb6295c5bef
Jul 06 18:56:05.278943 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.278872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2w7v7" event={"ID":"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716","Type":"ContainerStarted","Data":"2b6c3007af576d93be5563a04048987570cd2b3057f4efad3f85edb6295c5bef"}
Jul 06 18:56:05.614836 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.614791    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"]
Jul 06 18:56:05.618961 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.618916    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"]
Jul 06 18:56:05.619140 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.619061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.621996 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.621968    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.624535 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.624506    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"]
Jul 06 18:56:05.626080 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.626057    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 18:56:05.626347 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.626318    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 18:56:05.626617 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.626595    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 18:56:05.628071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.628050    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.628416 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.628395    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 06 18:56:05.631614 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.629224    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xp67v\""
Jul 06 18:56:05.631614 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.629295    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 18:56:05.635310 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.634611    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 18:56:05.635782 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.635760    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 18:56:05.637016 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.636084    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 18:56:05.637354 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.637322    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"]
Jul 06 18:56:05.643503 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.643445    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 18:56:05.650221 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.650187    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"]
Jul 06 18:56:05.661271 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.661244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"]
Jul 06 18:56:05.676123 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.675384    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:56:05.681351 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.681299    2561 patch_prober.go:28] interesting pod/image-registry-6bffd688bb-l594z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 06 18:56:05.681519 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.681369    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 18:56:05.700041 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-tmp\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.700220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700220 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700390 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.700390 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700523 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f022e9b5-d508-425e-991e-405fd92b5052-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.700523 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkmc\" (UniqueName: \"kubernetes.io/projected/a77dad21-0c30-40f2-a349-36fd3885b021-kube-api-access-lgkmc\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700523 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77dad21-0c30-40f2-a349-36fd3885b021-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.700679 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsrw\" (UniqueName: \"kubernetes.io/projected/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-kube-api-access-nhsrw\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.700679 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.700574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwk9\" (UniqueName: \"kubernetes.io/projected/f022e9b5-d508-425e-991e-405fd92b5052-kube-api-access-6hwk9\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.801584 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-tmp\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.801755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.801755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.801755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.801755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.801755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f022e9b5-d508-425e-991e-405fd92b5052-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkmc\" (UniqueName: \"kubernetes.io/projected/a77dad21-0c30-40f2-a349-36fd3885b021-kube-api-access-lgkmc\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77dad21-0c30-40f2-a349-36fd3885b021-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsrw\" (UniqueName: \"kubernetes.io/projected/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-kube-api-access-nhsrw\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801861    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwk9\" (UniqueName: \"kubernetes.io/projected/f022e9b5-d508-425e-991e-405fd92b5052-kube-api-access-6hwk9\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.802071 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.801947    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-tmp\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.803106 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.802965    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a77dad21-0c30-40f2-a349-36fd3885b021-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.806271 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.806241    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-ca\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.807068 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.807045    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.807907 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.807810    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-klusterlet-config\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.807907 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.807851    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.808402 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.808378    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a77dad21-0c30-40f2-a349-36fd3885b021-hub\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.808987 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.808964    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f022e9b5-d508-425e-991e-405fd92b5052-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.845559 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.836422    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwk9\" (UniqueName: \"kubernetes.io/projected/f022e9b5-d508-425e-991e-405fd92b5052-kube-api-access-6hwk9\") pod \"managed-serviceaccount-addon-agent-c568bc6cd-2vrj8\" (UID: \"f022e9b5-d508-425e-991e-405fd92b5052\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.845559 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.840235    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkmc\" (UniqueName: \"kubernetes.io/projected/a77dad21-0c30-40f2-a349-36fd3885b021-kube-api-access-lgkmc\") pod \"cluster-proxy-proxy-agent-c67657697-phqdq\" (UID: \"a77dad21-0c30-40f2-a349-36fd3885b021\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:05.845559 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.842043    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsrw\" (UniqueName: \"kubernetes.io/projected/cea3a424-a577-45a0-b34f-0c7f5ddaa8fa-kube-api-access-nhsrw\") pod \"klusterlet-addon-workmgr-56bbf8f7db-987zr\" (UID: \"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.853390 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.853358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7856c7b44-tnrlb"]
Jul 06 18:56:05.859926 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.859897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:05.919048 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.919010    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7856c7b44-tnrlb"]
Jul 06 18:56:05.946199 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.946162    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"
Jul 06 18:56:05.961123 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.961090    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:05.971974 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:05.971946    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"
Jul 06 18:56:06.003365 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-trusted-ca\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003543 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/485fee5f-4f28-4c04-bdcf-bf4d2905334e-ca-trust-extracted\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003543 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kxw\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-kube-api-access-64kxw\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003646 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-tls\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003646 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-image-registry-private-configuration\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003851 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-certificates\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003851 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-installation-pull-secrets\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.003851 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.003781    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-bound-sa-token\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-image-registry-private-configuration\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105702    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-certificates\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105732    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-installation-pull-secrets\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105773    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-bound-sa-token\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-trusted-ca\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105854    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/485fee5f-4f28-4c04-bdcf-bf4d2905334e-ca-trust-extracted\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kxw\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-kube-api-access-64kxw\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.105936    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-tls\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.107880    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-certificates\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.111108 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.108175    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/485fee5f-4f28-4c04-bdcf-bf4d2905334e-ca-trust-extracted\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.112604 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.112530    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/485fee5f-4f28-4c04-bdcf-bf4d2905334e-trusted-ca\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.116417 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.116367    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-registry-tls\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.119890 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.117938    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-installation-pull-secrets\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.119890 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.118910    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/485fee5f-4f28-4c04-bdcf-bf4d2905334e-image-registry-private-configuration\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.124342 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.124239    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kxw\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-kube-api-access-64kxw\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.124484 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.124396    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/485fee5f-4f28-4c04-bdcf-bf4d2905334e-bound-sa-token\") pod \"image-registry-7856c7b44-tnrlb\" (UID: \"485fee5f-4f28-4c04-bdcf-bf4d2905334e\") " pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.151979 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.151085    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8"]
Jul 06 18:56:06.156174 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:06.155494    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf022e9b5_d508_425e_991e_405fd92b5052.slice/crio-253d27a3d3d9ee2f1e4cd800640b34cc5869a0482b3a2f2cb2fee26fc54a26c0 WatchSource:0}: Error finding container 253d27a3d3d9ee2f1e4cd800640b34cc5869a0482b3a2f2cb2fee26fc54a26c0: Status 404 returned error can't find the container with id 253d27a3d3d9ee2f1e4cd800640b34cc5869a0482b3a2f2cb2fee26fc54a26c0
Jul 06 18:56:06.181328 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.181251    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"]
Jul 06 18:56:06.183979 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.183702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:06.185230 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:06.185194    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea3a424_a577_45a0_b34f_0c7f5ddaa8fa.slice/crio-f13a34a95adc6c27346417dc0b29e9ecc4939b0d595e739ed37cb7e718587124 WatchSource:0}: Error finding container f13a34a95adc6c27346417dc0b29e9ecc4939b0d595e739ed37cb7e718587124: Status 404 returned error can't find the container with id f13a34a95adc6c27346417dc0b29e9ecc4939b0d595e739ed37cb7e718587124
Jul 06 18:56:06.230850 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.230817    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq"]
Jul 06 18:56:06.235912 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:06.235881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda77dad21_0c30_40f2_a349_36fd3885b021.slice/crio-6c90c1f53a2882c4e9f5a0bf05fadc39ab3a5052616430cf3cb552e895406763 WatchSource:0}: Error finding container 6c90c1f53a2882c4e9f5a0bf05fadc39ab3a5052616430cf3cb552e895406763: Status 404 returned error can't find the container with id 6c90c1f53a2882c4e9f5a0bf05fadc39ab3a5052616430cf3cb552e895406763
Jul 06 18:56:06.285597 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.285514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq" event={"ID":"a77dad21-0c30-40f2-a349-36fd3885b021","Type":"ContainerStarted","Data":"6c90c1f53a2882c4e9f5a0bf05fadc39ab3a5052616430cf3cb552e895406763"}
Jul 06 18:56:06.289060 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.288950    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8" event={"ID":"f022e9b5-d508-425e-991e-405fd92b5052","Type":"ContainerStarted","Data":"253d27a3d3d9ee2f1e4cd800640b34cc5869a0482b3a2f2cb2fee26fc54a26c0"}
Jul 06 18:56:06.296017 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.295918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr" event={"ID":"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa","Type":"ContainerStarted","Data":"f13a34a95adc6c27346417dc0b29e9ecc4939b0d595e739ed37cb7e718587124"}
Jul 06 18:56:06.299938 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.299626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6f5c8" event={"ID":"b28c3344-93f0-4722-b2dd-c3a0e258839f","Type":"ContainerStarted","Data":"937e5a12c6cd306e11c40c0fd0ecfcf8efefb78358a13d316c9ede919d6d0a28"}
Jul 06 18:56:06.299938 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.299684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6f5c8" event={"ID":"b28c3344-93f0-4722-b2dd-c3a0e258839f","Type":"ContainerStarted","Data":"f54afa6e2995213206c8dd59e909be295e07fe99f9d156de1db948babe14d451"}
Jul 06 18:56:06.319680 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.317656    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6f5c8" podStartSLOduration=67.219230109 podStartE2EDuration="1m8.317633341s" totalImagesPullingTime="1.098403232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:06.317291054 +0000 UTC m=+68.903499028" watchObservedRunningTime="2026-07-06 18:56:06.317633341 +0000 UTC m=+68.903841316"
Jul 06 18:56:06.384230 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:06.382344    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7856c7b44-tnrlb"]
Jul 06 18:56:06.388593 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:06.388565    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485fee5f_4f28_4c04_bdcf_bf4d2905334e.slice/crio-c12899544021610d5e93e1e3174e0f002c41278551cb39fc39f3f9a822b8b553 WatchSource:0}: Error finding container c12899544021610d5e93e1e3174e0f002c41278551cb39fc39f3f9a822b8b553: Status 404 returned error can't find the container with id c12899544021610d5e93e1e3174e0f002c41278551cb39fc39f3f9a822b8b553
Jul 06 18:56:07.305991 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:07.305958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb" event={"ID":"485fee5f-4f28-4c04-bdcf-bf4d2905334e","Type":"ContainerStarted","Data":"7cca8195ce8891796898a3131bcb1ca2caf7d82350f06a00d7823e40e4dd454c"}
Jul 06 18:56:07.306480 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:07.306017    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb" event={"ID":"485fee5f-4f28-4c04-bdcf-bf4d2905334e","Type":"ContainerStarted","Data":"c12899544021610d5e93e1e3174e0f002c41278551cb39fc39f3f9a822b8b553"}
Jul 06 18:56:07.329504 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:07.329409    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb" podStartSLOduration=2.329385714 podStartE2EDuration="2.329385714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:07.328936121 +0000 UTC m=+69.915144096" watchObservedRunningTime="2026-07-06 18:56:07.329385714 +0000 UTC m=+69.915593690"
Jul 06 18:56:08.226658 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:08.226614    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:56:08.236434 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:08.236368    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6fccd601-0088-4800-864d-0c1b7bf65c64-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-5hnm4\" (UID: \"6fccd601-0088-4800-864d-0c1b7bf65c64\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:56:08.311497 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:08.311445    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:08.380369 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:08.380140    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-h7w9t\""
Jul 06 18:56:08.388451 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:08.388175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"
Jul 06 18:56:10.210294 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.210200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4"]
Jul 06 18:56:10.214575 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:10.214530    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fccd601_0088_4800_864d_0c1b7bf65c64.slice/crio-0f8b495b81adaf946782a8082429b8ab85c7c17833882d9382fb9264d509ffc3 WatchSource:0}: Error finding container 0f8b495b81adaf946782a8082429b8ab85c7c17833882d9382fb9264d509ffc3: Status 404 returned error can't find the container with id 0f8b495b81adaf946782a8082429b8ab85c7c17833882d9382fb9264d509ffc3
Jul 06 18:56:10.324529 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.324488    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2w7v7" event={"ID":"7c2f4853-e8c2-47f1-9ef4-f41e73e2e716","Type":"ContainerStarted","Data":"35a047f6a877ff9f49ee32d242dc9dddd4c41625bdbc9f23b8140e02cd18b03c"}
Jul 06 18:56:10.325344 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.325318    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:56:10.328007 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.327979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4" event={"ID":"6fccd601-0088-4800-864d-0c1b7bf65c64","Type":"ContainerStarted","Data":"0f8b495b81adaf946782a8082429b8ab85c7c17833882d9382fb9264d509ffc3"}
Jul 06 18:56:10.330697 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.330412    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-56wjc" event={"ID":"093150fe-2f19-46de-a8e3-714e7ee0779c","Type":"ContainerStarted","Data":"cb64b0ebb137211ceb14b423146d8faea368c8c45fd1cd83621d7f84e3df01e9"}
Jul 06 18:56:10.344034 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.343967    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2w7v7" podStartSLOduration=66.672222631 podStartE2EDuration="1m12.343941952s" totalImagesPullingTime="5.671719321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:10.343298163 +0000 UTC m=+72.929506139" watchObservedRunningTime="2026-07-06 18:56:10.343941952 +0000 UTC m=+72.930149927"
Jul 06 18:56:10.361822 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:10.361762    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-56wjc" podStartSLOduration=66.472024902 podStartE2EDuration="1m12.361744371s" totalImagesPullingTime="5.889719469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:10.360864613 +0000 UTC m=+72.947072587" watchObservedRunningTime="2026-07-06 18:56:10.361744371 +0000 UTC m=+72.947952347"
Jul 06 18:56:13.808176 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.808145    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2kv4"]
Jul 06 18:56:13.811852 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.811824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.815119 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.814684    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 18:56:13.815119 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.814961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s2prm\""
Jul 06 18:56:13.815735 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.815571    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 18:56:13.815735 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.815614    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 18:56:13.816114 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.816092    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 18:56:13.818755 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.818018    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 18:56:13.818871 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.818843    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 18:56:13.896886 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-textfile\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.896886 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-metrics-client-ca\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896940    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlr4p\" (UniqueName: \"kubernetes.io/projected/27b6e62d-d216-4718-8d93-778111bfd810-kube-api-access-vlr4p\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-root\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.896997    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.897021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-wtmp\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.897036    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.897124 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.897082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-sys\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998255 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-sys\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-textfile\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-metrics-client-ca\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998349    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-sys\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlr4p\" (UniqueName: \"kubernetes.io/projected/27b6e62d-d216-4718-8d93-778111bfd810-kube-api-access-vlr4p\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-root\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998752 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998491    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998752 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-wtmp\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.998752 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998557    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:56:13.998993    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:56:13.999065    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls podName:27b6e62d-d216-4718-8d93-778111bfd810 nodeName:}" failed. No retries permitted until 2026-07-06 18:56:14.499045847 +0000 UTC m=+77.085253810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls") pod "node-exporter-j2kv4" (UID: "27b6e62d-d216-4718-8d93-778111bfd810") : secret "node-exporter-tls" not found
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.999159    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.999290    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-wtmp\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.998996    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-textfile\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.999342 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.999304    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27b6e62d-d216-4718-8d93-778111bfd810-root\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:13.999732 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:13.999432    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27b6e62d-d216-4718-8d93-778111bfd810-metrics-client-ca\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:14.010286 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:14.010037    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:14.012929 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:14.012844    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlr4p\" (UniqueName: \"kubernetes.io/projected/27b6e62d-d216-4718-8d93-778111bfd810-kube-api-access-vlr4p\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:14.504445 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:14.504408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:14.507202 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:14.507176    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27b6e62d-d216-4718-8d93-778111bfd810-node-exporter-tls\") pod \"node-exporter-j2kv4\" (UID: \"27b6e62d-d216-4718-8d93-778111bfd810\") " pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:14.730272 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:14.730234    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2kv4"
Jul 06 18:56:15.063833 ip-10-0-138-123 kubenswrapper[2561]: W0706 18:56:15.063789    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b6e62d_d216_4718_8d93_778111bfd810.slice/crio-a070580ea1e57166cee31ac5159248d181290c0a01baa36825eda7bb2894a804 WatchSource:0}: Error finding container a070580ea1e57166cee31ac5159248d181290c0a01baa36825eda7bb2894a804: Status 404 returned error can't find the container with id a070580ea1e57166cee31ac5159248d181290c0a01baa36825eda7bb2894a804
Jul 06 18:56:15.352356 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.352315    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4" event={"ID":"6fccd601-0088-4800-864d-0c1b7bf65c64","Type":"ContainerStarted","Data":"6c46b32478e66875333c9a3f7199120703a3143d6140f9025f2dfaa91cff7db0"}
Jul 06 18:56:15.353805 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.353773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2kv4" event={"ID":"27b6e62d-d216-4718-8d93-778111bfd810","Type":"ContainerStarted","Data":"a070580ea1e57166cee31ac5159248d181290c0a01baa36825eda7bb2894a804"}
Jul 06 18:56:15.355507 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.355482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr" event={"ID":"cea3a424-a577-45a0-b34f-0c7f5ddaa8fa","Type":"ContainerStarted","Data":"46ba8f2324645719da40d8729da8b5a9f2a0cfe1c54a73bea901d6ef351016ca"}
Jul 06 18:56:15.355653 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.355623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:15.358002 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.357977    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr"
Jul 06 18:56:15.358121 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.358027    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq" event={"ID":"a77dad21-0c30-40f2-a349-36fd3885b021","Type":"ContainerStarted","Data":"5a13aa03c1bf0f98c0a5768a4ea43a72168e4deef75077a04d2dc3860f7d75c2"}
Jul 06 18:56:15.358121 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.358052    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq" event={"ID":"a77dad21-0c30-40f2-a349-36fd3885b021","Type":"ContainerStarted","Data":"da8e84f9a93d5a33e6c57cc1700b13214fda2955641cf7fa3d3e448334deba86"}
Jul 06 18:56:15.358121 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.358071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq" event={"ID":"a77dad21-0c30-40f2-a349-36fd3885b021","Type":"ContainerStarted","Data":"ef01b37cb70c29361e3719ce84e941c222e93fd16c49d8d9a873e01f967f8047"}
Jul 06 18:56:15.359643 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.359622    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8" event={"ID":"f022e9b5-d508-425e-991e-405fd92b5052","Type":"ContainerStarted","Data":"0e5cd20313b885e1961e613621cbe58fe7e039a7363c4760995ac0727fab5bff"}
Jul 06 18:56:15.373603 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.373547    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-5hnm4" podStartSLOduration=34.535708662 podStartE2EDuration="39.373527968s" totalImagesPullingTime="4.837819306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:15.373031176 +0000 UTC m=+77.959239200" watchObservedRunningTime="2026-07-06 18:56:15.373527968 +0000 UTC m=+77.959735941"
Jul 06 18:56:15.402859 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.402805    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56bbf8f7db-987zr" podStartSLOduration=1.515471384 podStartE2EDuration="10.402786191s" totalImagesPullingTime="8.887314807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:15.402409635 +0000 UTC m=+77.988617608" watchObservedRunningTime="2026-07-06 18:56:15.402786191 +0000 UTC m=+77.988994341"
Jul 06 18:56:15.428080 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.427995    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c568bc6cd-2vrj8" podStartSLOduration=1.530209143 podStartE2EDuration="10.427971557s" totalImagesPullingTime="8.897762414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:15.427636002 +0000 UTC m=+78.013843990" watchObservedRunningTime="2026-07-06 18:56:15.427971557 +0000 UTC m=+78.014179532"
Jul 06 18:56:15.454813 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.454759    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c67657697-phqdq" podStartSLOduration=1.62311635 podStartE2EDuration="10.454745044s" totalImagesPullingTime="8.831628694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:15.454029946 +0000 UTC m=+78.040237923" watchObservedRunningTime="2026-07-06 18:56:15.454745044 +0000 UTC m=+78.040953026"
Jul 06 18:56:15.680626 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:15.680599    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:56:16.368709 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:16.368668    2561 generic.go:356] "Generic (PLEG): container finished" podID="27b6e62d-d216-4718-8d93-778111bfd810" containerID="124637c3d129996694cbad2a64b5d1df1083ea475357847566faa7560f970a60" exitCode=0
Jul 06 18:56:16.369192 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:16.368768    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2kv4" event={"ID":"27b6e62d-d216-4718-8d93-778111bfd810","Type":"ContainerDied","Data":"124637c3d129996694cbad2a64b5d1df1083ea475357847566faa7560f970a60"}
Jul 06 18:56:17.375036 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:17.374995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2kv4" event={"ID":"27b6e62d-d216-4718-8d93-778111bfd810","Type":"ContainerStarted","Data":"78e55a0f308e46ffafd31c7c1849ecb9e159850e2ecdf575a2d45249a33dbce5"}
Jul 06 18:56:17.375442 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:17.375064    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2kv4" event={"ID":"27b6e62d-d216-4718-8d93-778111bfd810","Type":"ContainerStarted","Data":"10e8b5ddbe7dbe56bdfab634c294f706947b0c8a3a062ccc6f9fa5d0d665c18c"}
Jul 06 18:56:17.395703 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:17.395652    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2kv4" podStartSLOduration=3.26143407 podStartE2EDuration="4.395633571s" totalImagesPullingTime="1.062222057s" totalInitContainerRuntime="71.977444ms" isStatefulPod=true podCreationTimestamp="2026-07-06 18:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:56:17.393987807 +0000 UTC m=+79.980195788" watchObservedRunningTime="2026-07-06 18:56:17.395633571 +0000 UTC m=+79.981841544"
Jul 06 18:56:29.320960 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:29.320847    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7856c7b44-tnrlb"
Jul 06 18:56:30.701010 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:30.700967    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerName="registry" containerID="cri-o://38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356" gracePeriod=30
Jul 06 18:56:30.960320 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:30.960250    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:56:31.123389 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123389 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123401    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123428    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123492    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123517    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqf7t\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123539    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123657 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123569    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") pod \"014d0aa6-8fbd-461f-8dd7-48bad028504b\" (UID: \"014d0aa6-8fbd-461f-8dd7-48bad028504b\") "
Jul 06 18:56:31.123982 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123835    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 18:56:31.123982 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.123951    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 18:56:31.126340 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.126271    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 18:56:31.126644 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.126618    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 18:56:31.126741 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.126693    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t" (OuterVolumeSpecName: "kube-api-access-lqf7t") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "kube-api-access-lqf7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 18:56:31.126785 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.126754    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 18:56:31.126819 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.126796    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 18:56:31.135562 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.135525    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "014d0aa6-8fbd-461f-8dd7-48bad028504b" (UID: "014d0aa6-8fbd-461f-8dd7-48bad028504b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224053    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224083    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/014d0aa6-8fbd-461f-8dd7-48bad028504b-ca-trust-extracted\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224093    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-registry-certificates\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224107    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-image-registry-private-configuration\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224119    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/014d0aa6-8fbd-461f-8dd7-48bad028504b-trusted-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224127    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-bound-sa-token\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224135    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqf7t\" (UniqueName: \"kubernetes.io/projected/014d0aa6-8fbd-461f-8dd7-48bad028504b-kube-api-access-lqf7t\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.224137 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.224143    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/014d0aa6-8fbd-461f-8dd7-48bad028504b-installation-pull-secrets\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 18:56:31.431084 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.431047    2561 generic.go:356] "Generic (PLEG): container finished" podID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerID="38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356" exitCode=0
Jul 06 18:56:31.431264 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.431096    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6bffd688bb-l594z"
Jul 06 18:56:31.431264 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.431137    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" event={"ID":"014d0aa6-8fbd-461f-8dd7-48bad028504b","Type":"ContainerDied","Data":"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"}
Jul 06 18:56:31.431264 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.431205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6bffd688bb-l594z" event={"ID":"014d0aa6-8fbd-461f-8dd7-48bad028504b","Type":"ContainerDied","Data":"fe1735800309280c14c43faf24537fc9c5e9dc5949a3c7189095ec97638b8b1e"}
Jul 06 18:56:31.431264 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.431232    2561 scope.go:117] "RemoveContainer" containerID="38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"
Jul 06 18:56:31.440193 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.440176    2561 scope.go:117] "RemoveContainer" containerID="38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"
Jul 06 18:56:31.440489 ip-10-0-138-123 kubenswrapper[2561]: E0706 18:56:31.440449    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356\": container with ID starting with 38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356 not found: ID does not exist" containerID="38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"
Jul 06 18:56:31.440576 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.440498    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356"} err="failed to get container status \"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356\": rpc error: code = NotFound desc = could not find container \"38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356\": container with ID starting with 38d7cbdc99faf3693e1ab7f4894ca7a71cb2bf4ffde45a0b5741cd4e73b12356 not found: ID does not exist"
Jul 06 18:56:31.453776 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.453748    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:56:31.461854 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.461826    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6bffd688bb-l594z"]
Jul 06 18:56:31.961818 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:31.961790    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" path="/var/lib/kubelet/pods/014d0aa6-8fbd-461f-8dd7-48bad028504b/volumes"
Jul 06 18:56:42.341653 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:56:42.341623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2w7v7"
Jul 06 18:59:57.834362 ip-10-0-138-123 kubenswrapper[2561]: I0706 18:59:57.834331    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 19:01:01.407683 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.407599    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"]
Jul 06 19:01:01.408104 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.407926    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerName="registry"
Jul 06 19:01:01.408104 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.407938    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerName="registry"
Jul 06 19:01:01.408104 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.407989    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="014d0aa6-8fbd-461f-8dd7-48bad028504b" containerName="registry"
Jul 06 19:01:01.410795 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.410778    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.425920 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.425889    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-65wfm\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-65wfm\"" type="*v1.Secret"
Jul 06 19:01:01.436335 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.436300    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Jul 06 19:01:01.436335 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.436300    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 06 19:01:01.436540 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.436327    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="a4162355-478e-47b2-b65b-e9c652a13074" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.436540 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.436345    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 06 19:01:01.436540 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:01.436345    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Jul 06 19:01:01.498696 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.498658    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"]
Jul 06 19:01:01.556768 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.556734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspv2\" (UniqueName: \"kubernetes.io/projected/a4162355-478e-47b2-b65b-e9c652a13074-kube-api-access-jspv2\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.556944 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.556786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-manifests\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.556944 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.556854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-tmp\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.556944 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.556871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.556944 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.556898    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658152    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-manifests\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-tmp\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658216    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658238    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658371    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspv2\" (UniqueName: \"kubernetes.io/projected/a4162355-478e-47b2-b65b-e9c652a13074-kube-api-access-jspv2\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.658691 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.658651    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-manifests\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:01.660665 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:01.660645    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4162355-478e-47b2-b65b-e9c652a13074-tmp\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:02.515972 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.515943    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 19:01:02.594154 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.594121    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 19:01:02.658874 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:02.658829    2561 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 06 19:01:02.659060 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:02.658926    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert podName:a4162355-478e-47b2-b65b-e9c652a13074 nodeName:}" failed. No retries permitted until 2026-07-06 19:01:03.158907775 +0000 UTC m=+365.745115727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert") pod "opendatahub-operator-controller-manager-546c6c6d78-zdx6s" (UID: "a4162355-478e-47b2-b65b-e9c652a13074") : failed to sync secret cache: timed out waiting for the condition
Jul 06 19:01:02.659060 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:02.658830    2561 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition
Jul 06 19:01:02.659060 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:02.659011    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert podName:a4162355-478e-47b2-b65b-e9c652a13074 nodeName:}" failed. No retries permitted until 2026-07-06 19:01:03.158991291 +0000 UTC m=+365.745199243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert") pod "opendatahub-operator-controller-manager-546c6c6d78-zdx6s" (UID: "a4162355-478e-47b2-b65b-e9c652a13074") : failed to sync secret cache: timed out waiting for the condition
Jul 06 19:01:02.689284 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.689256    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 19:01:02.697962 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.697938    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspv2\" (UniqueName: \"kubernetes.io/projected/a4162355-478e-47b2-b65b-e9c652a13074-kube-api-access-jspv2\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:02.781196 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.781114    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-65wfm\""
Jul 06 19:01:02.816012 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:02.815987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 19:01:03.170998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.170961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:03.170998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.171004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:03.173581 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.173550    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-webhook-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:03.173704 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.173640    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4162355-478e-47b2-b65b-e9c652a13074-apiservice-cert\") pod \"opendatahub-operator-controller-manager-546c6c6d78-zdx6s\" (UID: \"a4162355-478e-47b2-b65b-e9c652a13074\") " pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:03.220666 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.220621    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:03.364193 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.364165    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"]
Jul 06 19:01:03.368423 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:01:03.368396    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4162355_478e_47b2_b65b_e9c652a13074.slice/crio-3c6beb69653ad31dcec0f66a3e9756c42daaf6f6eb6bd4c94e905f2e336575b8 WatchSource:0}: Error finding container 3c6beb69653ad31dcec0f66a3e9756c42daaf6f6eb6bd4c94e905f2e336575b8: Status 404 returned error can't find the container with id 3c6beb69653ad31dcec0f66a3e9756c42daaf6f6eb6bd4c94e905f2e336575b8
Jul 06 19:01:03.370298 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:03.370280    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:01:04.358989 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:04.358956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s" event={"ID":"a4162355-478e-47b2-b65b-e9c652a13074","Type":"ContainerStarted","Data":"3c6beb69653ad31dcec0f66a3e9756c42daaf6f6eb6bd4c94e905f2e336575b8"}
Jul 06 19:01:06.368901 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:06.368867    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s" event={"ID":"a4162355-478e-47b2-b65b-e9c652a13074","Type":"ContainerStarted","Data":"b0c3b23a3d07c66aa0b64fefce7a84736d642ded36d91da24065e6da0ef1d298"}
Jul 06 19:01:07.375734 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:07.375701    2561 generic.go:356] "Generic (PLEG): container finished" podID="a4162355-478e-47b2-b65b-e9c652a13074" containerID="b0c3b23a3d07c66aa0b64fefce7a84736d642ded36d91da24065e6da0ef1d298" exitCode=0
Jul 06 19:01:07.376106 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:07.375795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s" event={"ID":"a4162355-478e-47b2-b65b-e9c652a13074","Type":"ContainerDied","Data":"b0c3b23a3d07c66aa0b64fefce7a84736d642ded36d91da24065e6da0ef1d298"}
Jul 06 19:01:08.381014 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:08.380976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s" event={"ID":"a4162355-478e-47b2-b65b-e9c652a13074","Type":"ContainerStarted","Data":"4106bb7fe3e1579eb4d19bd213d449b80db98810dc773bf0edcf75f270a2aa9f"}
Jul 06 19:01:08.381489 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:08.381103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:19.387435 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:19.387403    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s"
Jul 06 19:01:19.424619 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:19.424551    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-546c6c6d78-zdx6s" podStartSLOduration=14.302112495 podStartE2EDuration="18.424534075s" totalImagesPullingTime="3.074094355s" totalInitContainerRuntime="1.048327225s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:01:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:01:08.42058114 +0000 UTC m=+371.006789125" watchObservedRunningTime="2026-07-06 19:01:19.424534075 +0000 UTC m=+382.010742048"
Jul 06 19:01:31.476289 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.476254    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"]
Jul 06 19:01:31.482139 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.482116    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.486878 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.486850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 06 19:01:31.487029 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.486850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 06 19:01:31.487029 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.486920    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nmr7s\""
Jul 06 19:01:31.487156 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.487118    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 06 19:01:31.487156 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.487136    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 06 19:01:31.487244 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.487183    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 06 19:01:31.497643 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.497617    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"]
Jul 06 19:01:31.500733 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.500707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7574516d-6feb-423f-8143-0609aaceed05-manager-config\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.500850 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.500737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.500850 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.500811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-metrics-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.500850 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.500841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvf6k\" (UniqueName: \"kubernetes.io/projected/7574516d-6feb-423f-8143-0609aaceed05-kube-api-access-mvf6k\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.601792 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.601752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-metrics-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.602072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.602053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvf6k\" (UniqueName: \"kubernetes.io/projected/7574516d-6feb-423f-8143-0609aaceed05-kube-api-access-mvf6k\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.602233 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.602219    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7574516d-6feb-423f-8143-0609aaceed05-manager-config\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.602330 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.602318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.603242 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.603186    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7574516d-6feb-423f-8143-0609aaceed05-manager-config\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.606072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.606017    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-metrics-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.608539 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.608510    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7574516d-6feb-423f-8143-0609aaceed05-cert\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.616423 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.616382    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvf6k\" (UniqueName: \"kubernetes.io/projected/7574516d-6feb-423f-8143-0609aaceed05-kube-api-access-mvf6k\") pod \"lws-controller-manager-5d5c698c7c-t6jpf\" (UID: \"7574516d-6feb-423f-8143-0609aaceed05\") " pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.791053 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.790956    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:31.955189 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:31.955155    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"]
Jul 06 19:01:31.959032 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:01:31.958999    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7574516d_6feb_423f_8143_0609aaceed05.slice/crio-e24a3bd4fe5a6970d35f8a7e407bb190c9f7973dd9807da3110b5800901549ab WatchSource:0}: Error finding container e24a3bd4fe5a6970d35f8a7e407bb190c9f7973dd9807da3110b5800901549ab: Status 404 returned error can't find the container with id e24a3bd4fe5a6970d35f8a7e407bb190c9f7973dd9807da3110b5800901549ab
Jul 06 19:01:32.467042 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:32.467004    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf" event={"ID":"7574516d-6feb-423f-8143-0609aaceed05","Type":"ContainerStarted","Data":"e24a3bd4fe5a6970d35f8a7e407bb190c9f7973dd9807da3110b5800901549ab"}
Jul 06 19:01:35.481393 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.481360    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf" event={"ID":"7574516d-6feb-423f-8143-0609aaceed05","Type":"ContainerStarted","Data":"d7a196a2cbbb0593e07900a432b005a4c40798486860347b9adfc6b1ea56c603"}
Jul 06 19:01:35.481811 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.481495    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:01:35.711786 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.711734    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf" podStartSLOduration=2.081097494 podStartE2EDuration="4.711713751s" totalImagesPullingTime="2.630616257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:01:35.71072759 +0000 UTC m=+398.296935576" watchObservedRunningTime="2026-07-06 19:01:35.711713751 +0000 UTC m=+398.297921768"
Jul 06 19:01:35.936160 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.936121    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"]
Jul 06 19:01:35.939739 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.939716    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:35.944948 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.944926    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 06 19:01:35.945475 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.945443    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-x54xh\""
Jul 06 19:01:35.968976 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:35.968945    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"]
Jul 06 19:01:36.039271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.039237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.039504 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.039300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf26\" (UniqueName: \"kubernetes.io/projected/5b572ebf-05e3-48a2-b944-580b6571dc04-kube-api-access-txf26\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.140578 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.140546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf26\" (UniqueName: \"kubernetes.io/projected/5b572ebf-05e3-48a2-b944-580b6571dc04-kube-api-access-txf26\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.140750 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.140609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.140750 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:36.140712    2561 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 06 19:01:36.140824 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:01:36.140772    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert podName:5b572ebf-05e3-48a2-b944-580b6571dc04 nodeName:}" failed. No retries permitted until 2026-07-06 19:01:36.640753536 +0000 UTC m=+399.226961489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert") pod "llmisvc-controller-manager-777c6bd7d5-q52mx" (UID: "5b572ebf-05e3-48a2-b944-580b6571dc04") : secret "llmisvc-webhook-server-cert" not found
Jul 06 19:01:36.172401 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.172369    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf26\" (UniqueName: \"kubernetes.io/projected/5b572ebf-05e3-48a2-b944-580b6571dc04-kube-api-access-txf26\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.645151 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.645111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.647659 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.647640    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b572ebf-05e3-48a2-b944-580b6571dc04-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q52mx\" (UID: \"5b572ebf-05e3-48a2-b944-580b6571dc04\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:36.851210 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:36.851158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:37.010281 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:37.010244    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"]
Jul 06 19:01:37.013640 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:01:37.013607    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b572ebf_05e3_48a2_b944_580b6571dc04.slice/crio-9aa6f5eada93b78d8cfaf6b0c9c8b25bf2aa25f7f19d5a7891264744e5d405ba WatchSource:0}: Error finding container 9aa6f5eada93b78d8cfaf6b0c9c8b25bf2aa25f7f19d5a7891264744e5d405ba: Status 404 returned error can't find the container with id 9aa6f5eada93b78d8cfaf6b0c9c8b25bf2aa25f7f19d5a7891264744e5d405ba
Jul 06 19:01:37.489809 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:37.489777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx" event={"ID":"5b572ebf-05e3-48a2-b944-580b6571dc04","Type":"ContainerStarted","Data":"9aa6f5eada93b78d8cfaf6b0c9c8b25bf2aa25f7f19d5a7891264744e5d405ba"}
Jul 06 19:01:42.510648 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:42.510614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx" event={"ID":"5b572ebf-05e3-48a2-b944-580b6571dc04","Type":"ContainerStarted","Data":"1b9f8a9b261e9d200f270e5c482949f93045bd6a7d014b493a4bee34e19efc31"}
Jul 06 19:01:42.511072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:42.510759    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:01:42.540114 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:42.540061    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx" podStartSLOduration=2.795306554 podStartE2EDuration="7.540042682s" totalImagesPullingTime="4.744736128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:01:42.538827953 +0000 UTC m=+405.125035962" watchObservedRunningTime="2026-07-06 19:01:42.540042682 +0000 UTC m=+405.126250657"
Jul 06 19:01:46.487991 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:01:46.487958    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d5c698c7c-t6jpf"
Jul 06 19:02:13.516626 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:13.516596    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q52mx"
Jul 06 19:02:23.044605 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.044518    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:23.047848 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.047820    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:23.050662 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.050639    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 19:02:23.050839 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.050773    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-snhpd\""
Jul 06 19:02:23.051518 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.051497    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 19:02:23.062124 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.062070    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:23.208053 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.208018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz2m\" (UniqueName: \"kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m\") pod \"kuadrant-operator-catalog-8sjxp\" (UID: \"7db53152-8758-484e-a46d-24a195f9fa69\") " pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:23.309110 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.309081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz2m\" (UniqueName: \"kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m\") pod \"kuadrant-operator-catalog-8sjxp\" (UID: \"7db53152-8758-484e-a46d-24a195f9fa69\") " pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:23.319500 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.319450    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz2m\" (UniqueName: \"kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m\") pod \"kuadrant-operator-catalog-8sjxp\" (UID: \"7db53152-8758-484e-a46d-24a195f9fa69\") " pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:23.359278 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.359230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:23.397217 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.397184    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:23.499083 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.499051    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:23.502230 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:02:23.502200    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db53152_8758_484e_a46d_24a195f9fa69.slice/crio-0504ed1b62a0e38a1b5d53ac72c1ba452673476fa84273935e9bf20cd7dd0094 WatchSource:0}: Error finding container 0504ed1b62a0e38a1b5d53ac72c1ba452673476fa84273935e9bf20cd7dd0094: Status 404 returned error can't find the container with id 0504ed1b62a0e38a1b5d53ac72c1ba452673476fa84273935e9bf20cd7dd0094
Jul 06 19:02:23.615632 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.615547    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7m9s4"]
Jul 06 19:02:23.619934 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.619916    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:23.633686 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.633658    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7m9s4"]
Jul 06 19:02:23.664727 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.664685    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" event={"ID":"7db53152-8758-484e-a46d-24a195f9fa69","Type":"ContainerStarted","Data":"0504ed1b62a0e38a1b5d53ac72c1ba452673476fa84273935e9bf20cd7dd0094"}
Jul 06 19:02:23.713128 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.713092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqbp9\" (UniqueName: \"kubernetes.io/projected/c2a51fec-a938-479c-972a-108694e4d5c7-kube-api-access-gqbp9\") pod \"kuadrant-operator-catalog-7m9s4\" (UID: \"c2a51fec-a938-479c-972a-108694e4d5c7\") " pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:23.813810 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.813774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqbp9\" (UniqueName: \"kubernetes.io/projected/c2a51fec-a938-479c-972a-108694e4d5c7-kube-api-access-gqbp9\") pod \"kuadrant-operator-catalog-7m9s4\" (UID: \"c2a51fec-a938-479c-972a-108694e4d5c7\") " pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:23.822220 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.822190    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqbp9\" (UniqueName: \"kubernetes.io/projected/c2a51fec-a938-479c-972a-108694e4d5c7-kube-api-access-gqbp9\") pod \"kuadrant-operator-catalog-7m9s4\" (UID: \"c2a51fec-a938-479c-972a-108694e4d5c7\") " pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:23.934275 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:23.934177    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:24.079936 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:24.079902    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7m9s4"]
Jul 06 19:02:24.083356 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:02:24.083301    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a51fec_a938_479c_972a_108694e4d5c7.slice/crio-d2a2a4db11b06baf7e75d11c6c62b7983cdf255a41b4805e12c48b4471219d8d WatchSource:0}: Error finding container d2a2a4db11b06baf7e75d11c6c62b7983cdf255a41b4805e12c48b4471219d8d: Status 404 returned error can't find the container with id d2a2a4db11b06baf7e75d11c6c62b7983cdf255a41b4805e12c48b4471219d8d
Jul 06 19:02:24.672486 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:24.672435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4" event={"ID":"c2a51fec-a938-479c-972a-108694e4d5c7","Type":"ContainerStarted","Data":"d2a2a4db11b06baf7e75d11c6c62b7983cdf255a41b4805e12c48b4471219d8d"}
Jul 06 19:02:26.682696 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.682652    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" event={"ID":"7db53152-8758-484e-a46d-24a195f9fa69","Type":"ContainerStarted","Data":"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"}
Jul 06 19:02:26.683139 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.682704    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" podUID="7db53152-8758-484e-a46d-24a195f9fa69" containerName="registry-server" containerID="cri-o://7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680" gracePeriod=2
Jul 06 19:02:26.684357 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.684330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4" event={"ID":"c2a51fec-a938-479c-972a-108694e4d5c7","Type":"ContainerStarted","Data":"c3433ee507763f24b7d31a8a698f57c10f538019cf7a864e40143076f5f2fdd3"}
Jul 06 19:02:26.700127 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.700082    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" podStartSLOduration=1.515561296 podStartE2EDuration="3.70006568s" totalImagesPullingTime="2.184504384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:02:26.699161163 +0000 UTC m=+449.285369137" watchObservedRunningTime="2026-07-06 19:02:26.70006568 +0000 UTC m=+449.286273655"
Jul 06 19:02:26.715061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.715014    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4" podStartSLOduration=2.109164067 podStartE2EDuration="3.7149971s" totalImagesPullingTime="1.605833033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:02:26.714566726 +0000 UTC m=+449.300774718" watchObservedRunningTime="2026-07-06 19:02:26.7149971 +0000 UTC m=+449.301205074"
Jul 06 19:02:26.934859 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:26.934794    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:27.042112 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.042071    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxz2m\" (UniqueName: \"kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m\") pod \"7db53152-8758-484e-a46d-24a195f9fa69\" (UID: \"7db53152-8758-484e-a46d-24a195f9fa69\") "
Jul 06 19:02:27.044495 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.044439    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m" (OuterVolumeSpecName: "kube-api-access-fxz2m") pod "7db53152-8758-484e-a46d-24a195f9fa69" (UID: "7db53152-8758-484e-a46d-24a195f9fa69"). InnerVolumeSpecName "kube-api-access-fxz2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:02:27.142643 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.142607    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxz2m\" (UniqueName: \"kubernetes.io/projected/7db53152-8758-484e-a46d-24a195f9fa69-kube-api-access-fxz2m\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:02:27.690050 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.690012    2561 generic.go:356] "Generic (PLEG): container finished" podID="7db53152-8758-484e-a46d-24a195f9fa69" containerID="7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680" exitCode=0
Jul 06 19:02:27.690542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.690070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp"
Jul 06 19:02:27.690542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.690096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" event={"ID":"7db53152-8758-484e-a46d-24a195f9fa69","Type":"ContainerDied","Data":"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"}
Jul 06 19:02:27.690542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.690167    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8sjxp" event={"ID":"7db53152-8758-484e-a46d-24a195f9fa69","Type":"ContainerDied","Data":"0504ed1b62a0e38a1b5d53ac72c1ba452673476fa84273935e9bf20cd7dd0094"}
Jul 06 19:02:27.690542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.690187    2561 scope.go:117] "RemoveContainer" containerID="7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"
Jul 06 19:02:27.699076 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.699056    2561 scope.go:117] "RemoveContainer" containerID="7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"
Jul 06 19:02:27.699408 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:02:27.699390    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680\": container with ID starting with 7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680 not found: ID does not exist" containerID="7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"
Jul 06 19:02:27.699490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.699418    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680"} err="failed to get container status \"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680\": rpc error: code = NotFound desc = could not find container \"7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680\": container with ID starting with 7a959db8952905b6f54a117185240363e7039cb95b3e340a655bb4047b481680 not found: ID does not exist"
Jul 06 19:02:27.718127 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.718090    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:27.721868 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.721842    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8sjxp"]
Jul 06 19:02:27.962112 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:27.962032    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db53152-8758-484e-a46d-24a195f9fa69" path="/var/lib/kubelet/pods/7db53152-8758-484e-a46d-24a195f9fa69/volumes"
Jul 06 19:02:33.934513 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:33.934454    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:33.935075 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:33.934552    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:33.961920 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:33.961894    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:02:34.740626 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:02:34.740598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7m9s4"
Jul 06 19:03:01.141947 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.141904    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"]
Jul 06 19:03:01.142329 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.142185    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db53152-8758-484e-a46d-24a195f9fa69" containerName="registry-server"
Jul 06 19:03:01.142329 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.142197    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db53152-8758-484e-a46d-24a195f9fa69" containerName="registry-server"
Jul 06 19:03:01.142329 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.142248    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db53152-8758-484e-a46d-24a195f9fa69" containerName="registry-server"
Jul 06 19:03:01.144482 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.144446    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.147335 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.147320    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7rhkt\""
Jul 06 19:03:01.163789 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.163755    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"]
Jul 06 19:03:01.216513 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.216455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.216701 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.216526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n785d\" (UniqueName: \"kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.317542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.317506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.317542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.317554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n785d\" (UniqueName: \"kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.317934 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.317909    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.329588 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.329563    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n785d\" (UniqueName: \"kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.455045 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.454962    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:01.596188 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.596052    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"]
Jul 06 19:03:01.599183 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:03:01.599146    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b49117_905d_4310_b692_b08ae3e4f0ee.slice/crio-cce0feb7ee8ecdae46825a1f85e1ffa4e252830c3c8ea42ce530ca375581b512 WatchSource:0}: Error finding container cce0feb7ee8ecdae46825a1f85e1ffa4e252830c3c8ea42ce530ca375581b512: Status 404 returned error can't find the container with id cce0feb7ee8ecdae46825a1f85e1ffa4e252830c3c8ea42ce530ca375581b512
Jul 06 19:03:01.822145 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:01.822107    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6" event={"ID":"24b49117-905d-4310-b692-b08ae3e4f0ee","Type":"ContainerStarted","Data":"cce0feb7ee8ecdae46825a1f85e1ffa4e252830c3c8ea42ce530ca375581b512"}
Jul 06 19:03:08.853126 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:08.853090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6" event={"ID":"24b49117-905d-4310-b692-b08ae3e4f0ee","Type":"ContainerStarted","Data":"94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be"}
Jul 06 19:03:08.853574 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:08.853230    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:08.885951 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:08.885897    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6" podStartSLOduration=1.353976047 podStartE2EDuration="7.885877156s" totalImagesPullingTime="6.531901109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:03:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:03:08.884647097 +0000 UTC m=+491.470855090" watchObservedRunningTime="2026-07-06 19:03:08.885877156 +0000 UTC m=+491.472085129"
Jul 06 19:03:10.893857 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.893815    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"]
Jul 06 19:03:10.897351 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.897326    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:10.902674 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.902648    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nqzkc\""
Jul 06 19:03:10.902813 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.902763    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 06 19:03:10.903225 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.903201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 06 19:03:10.906327 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.906304    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"]
Jul 06 19:03:10.997911 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.997875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a89d790-aa8a-47e6-89f8-507605ad13ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:10.997911 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.997931    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a89d790-aa8a-47e6-89f8-507605ad13ba-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:10.998143 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:10.997971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjb7\" (UniqueName: \"kubernetes.io/projected/9a89d790-aa8a-47e6-89f8-507605ad13ba-kube-api-access-sgjb7\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.099423 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.099386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a89d790-aa8a-47e6-89f8-507605ad13ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.099627 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.099446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a89d790-aa8a-47e6-89f8-507605ad13ba-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.099627 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.099501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjb7\" (UniqueName: \"kubernetes.io/projected/9a89d790-aa8a-47e6-89f8-507605ad13ba-kube-api-access-sgjb7\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.100168 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.100141    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a89d790-aa8a-47e6-89f8-507605ad13ba-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.102022 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.101993    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a89d790-aa8a-47e6-89f8-507605ad13ba-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.107902 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.107880    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjb7\" (UniqueName: \"kubernetes.io/projected/9a89d790-aa8a-47e6-89f8-507605ad13ba-kube-api-access-sgjb7\") pod \"kuadrant-console-plugin-66b5774985-cbfm8\" (UID: \"9a89d790-aa8a-47e6-89f8-507605ad13ba\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.207119 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.207023    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"
Jul 06 19:03:11.358583 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.358547    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8"]
Jul 06 19:03:11.361614 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:03:11.361580    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a89d790_aa8a_47e6_89f8_507605ad13ba.slice/crio-ebae74932439a4f3dd67d5a6b551c15179896250e0edd14ece96f7b6dd23f871 WatchSource:0}: Error finding container ebae74932439a4f3dd67d5a6b551c15179896250e0edd14ece96f7b6dd23f871: Status 404 returned error can't find the container with id ebae74932439a4f3dd67d5a6b551c15179896250e0edd14ece96f7b6dd23f871
Jul 06 19:03:11.867597 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:11.867557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8" event={"ID":"9a89d790-aa8a-47e6-89f8-507605ad13ba","Type":"ContainerStarted","Data":"ebae74932439a4f3dd67d5a6b551c15179896250e0edd14ece96f7b6dd23f871"}
Jul 06 19:03:19.859174 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:19.859145    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:22.109619 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.109580    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"]
Jul 06 19:03:22.110105 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.109817    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" containerName="manager" containerID="cri-o://94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be" gracePeriod=2
Jul 06 19:03:22.126110 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.126070    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"]
Jul 06 19:03:22.130595 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:22.130558    2561 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"24b49117-905d-4310-b692-b08ae3e4f0ee\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:22.161615 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.161578    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:22.161993 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.161978    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" containerName="manager"
Jul 06 19:03:22.162071 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.161998    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" containerName="manager"
Jul 06 19:03:22.162131 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.162091    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" containerName="manager"
Jul 06 19:03:22.173134 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.173100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.178029 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:22.177985    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:22.191783 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.191756    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:22.294287 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.294244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlnz\" (UniqueName: \"kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.294506 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.294434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.395165 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.395073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.395165 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.395117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlnz\" (UniqueName: \"kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.395549 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.395531    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.428574 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.428536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlnz\" (UniqueName: \"kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jvjs7\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:22.496672 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:22.496627    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:23.913617 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:23.913576    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:03:23.933527 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:23.933354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:23.991512 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:23.991479    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:03:24.011002 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.010970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.011180 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.011054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47vj\" (UniqueName: \"kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.064628 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:24.064585    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:24.111955 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.111917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47vj\" (UniqueName: \"kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.112337 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.112303    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.112777 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.112754    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.130402 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.130367    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47vj\" (UniqueName: \"kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj\") pod \"kuadrant-operator-controller-manager-86878c769f-4scfx\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:24.247392 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:24.247303    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:34.637344 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.637282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:03:34.639381 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:03:34.639350    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fcfdc9_459e_4ce6_a1f4_657aaa73313b.slice/crio-85d8a72e5f22b4b9aba76dac7b35a2c626d6a75ade3abc316cf456c2872b7cf2 WatchSource:0}: Error finding container 85d8a72e5f22b4b9aba76dac7b35a2c626d6a75ade3abc316cf456c2872b7cf2: Status 404 returned error can't find the container with id 85d8a72e5f22b4b9aba76dac7b35a2c626d6a75ade3abc316cf456c2872b7cf2
Jul 06 19:03:34.646429 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.646406    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:34.648554 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:34.648515    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:34.670594 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.670435    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:34.673769 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:03:34.673740    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d6eba2_cd6d_48a4_8024_e490f2f935a2.slice/crio-66fed453ada6eebb91beb1f30810272d3f0aa83873568358b95d2019a18f64df WatchSource:0}: Error finding container 66fed453ada6eebb91beb1f30810272d3f0aa83873568358b95d2019a18f64df: Status 404 returned error can't find the container with id 66fed453ada6eebb91beb1f30810272d3f0aa83873568358b95d2019a18f64df
Jul 06 19:03:34.809603 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.809567    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n785d\" (UniqueName: \"kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d\") pod \"24b49117-905d-4310-b692-b08ae3e4f0ee\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") "
Jul 06 19:03:34.809802 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.809625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume\") pod \"24b49117-905d-4310-b692-b08ae3e4f0ee\" (UID: \"24b49117-905d-4310-b692-b08ae3e4f0ee\") "
Jul 06 19:03:34.810136 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.810108    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24b49117-905d-4310-b692-b08ae3e4f0ee" (UID: "24b49117-905d-4310-b692-b08ae3e4f0ee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:03:34.812087 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.812063    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d" (OuterVolumeSpecName: "kube-api-access-n785d") pod "24b49117-905d-4310-b692-b08ae3e4f0ee" (UID: "24b49117-905d-4310-b692-b08ae3e4f0ee"). InnerVolumeSpecName "kube-api-access-n785d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:03:34.910365 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.910321    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n785d\" (UniqueName: \"kubernetes.io/projected/24b49117-905d-4310-b692-b08ae3e4f0ee-kube-api-access-n785d\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:03:34.910365 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.910358    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24b49117-905d-4310-b692-b08ae3e4f0ee-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:03:34.977159 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.977062    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8" event={"ID":"9a89d790-aa8a-47e6-89f8-507605ad13ba","Type":"ContainerStarted","Data":"e9f58ad314261997ca47c707a0fee1e2829bf368a0b3eaa81834977aff9a31b8"}
Jul 06 19:03:34.978735 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.978710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" event={"ID":"79fcfdc9-459e-4ce6-a1f4-657aaa73313b","Type":"ContainerStarted","Data":"af8257ff083a6b2766d6973368777537c366b4a3e63c65ab58f66115b6c6a403"}
Jul 06 19:03:34.978854 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.978756    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" event={"ID":"79fcfdc9-459e-4ce6-a1f4-657aaa73313b","Type":"ContainerStarted","Data":"85d8a72e5f22b4b9aba76dac7b35a2c626d6a75ade3abc316cf456c2872b7cf2"}
Jul 06 19:03:34.978854 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.978806    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:34.979937 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:34.979904    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:34.980407 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.980385    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" event={"ID":"f0d6eba2-cd6d-48a4-8024-e490f2f935a2","Type":"ContainerStarted","Data":"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"}
Jul 06 19:03:34.980512 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.980426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" event={"ID":"f0d6eba2-cd6d-48a4-8024-e490f2f935a2","Type":"ContainerStarted","Data":"66fed453ada6eebb91beb1f30810272d3f0aa83873568358b95d2019a18f64df"}
Jul 06 19:03:34.980581 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.980518    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:34.981543 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.981518    2561 generic.go:356] "Generic (PLEG): container finished" podID="24b49117-905d-4310-b692-b08ae3e4f0ee" containerID="94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be" exitCode=0
Jul 06 19:03:34.981630 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.981557    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:34.981630 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.981579    2561 scope.go:117] "RemoveContainer" containerID="94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be"
Jul 06 19:03:34.989874 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.989859    2561 scope.go:117] "RemoveContainer" containerID="94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be"
Jul 06 19:03:34.990151 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:34.990131    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be\": container with ID starting with 94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be not found: ID does not exist" containerID="94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be"
Jul 06 19:03:34.990193 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:34.990160    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be"} err="failed to get container status \"94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be\": rpc error: code = NotFound desc = could not find container \"94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be\": container with ID starting with 94809b485e2ee7b87970ceaceac8a7f9a6734d0418b6327cd620f478a710f8be not found: ID does not exist"
Jul 06 19:03:35.006369 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:35.006316    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cbfm8" podStartSLOduration=1.817860178 podStartE2EDuration="25.006299916s" totalImagesPullingTime="23.188439738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:03:35.004308854 +0000 UTC m=+517.590516828" watchObservedRunningTime="2026-07-06 19:03:35.006299916 +0000 UTC m=+517.592507916"
Jul 06 19:03:35.006614 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:35.006588    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-sd4b6\" is forbidden: User \"system:node:ip-10-0-138-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-123.ec2.internal' and this object" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sd4b6"
Jul 06 19:03:35.035561 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:35.034152    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" podStartSLOduration=13.034128933 podStartE2EDuration="13.034128933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:03:35.033694942 +0000 UTC m=+517.619902927" watchObservedRunningTime="2026-07-06 19:03:35.034128933 +0000 UTC m=+517.620336908"
Jul 06 19:03:35.065504 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:35.065423    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" podStartSLOduration=12.065404485 podStartE2EDuration="12.065404485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:03:35.064021947 +0000 UTC m=+517.650229959" watchObservedRunningTime="2026-07-06 19:03:35.065404485 +0000 UTC m=+517.651612458"
Jul 06 19:03:35.964790 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:35.964754    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b49117-905d-4310-b692-b08ae3e4f0ee" path="/var/lib/kubelet/pods/24b49117-905d-4310-b692-b08ae3e4f0ee/volumes"
Jul 06 19:03:45.989006 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:45.988971    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:03:45.989633 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:45.989054    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:46.073600 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.073564    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:46.073793 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.073771    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" podUID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" containerName="manager" containerID="cri-o://342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256" gracePeriod=10
Jul 06 19:03:46.333144 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.333113    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:46.390259 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.390222    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume\") pod \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") "
Jul 06 19:03:46.390449 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.390285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdlnz\" (UniqueName: \"kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz\") pod \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\" (UID: \"f0d6eba2-cd6d-48a4-8024-e490f2f935a2\") "
Jul 06 19:03:46.390690 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.390667    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0d6eba2-cd6d-48a4-8024-e490f2f935a2" (UID: "f0d6eba2-cd6d-48a4-8024-e490f2f935a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:03:46.392672 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.392648    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz" (OuterVolumeSpecName: "kube-api-access-kdlnz") pod "f0d6eba2-cd6d-48a4-8024-e490f2f935a2" (UID: "f0d6eba2-cd6d-48a4-8024-e490f2f935a2"). InnerVolumeSpecName "kube-api-access-kdlnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:03:46.491134 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.491102    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:03:46.491134 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:46.491128    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlnz\" (UniqueName: \"kubernetes.io/projected/f0d6eba2-cd6d-48a4-8024-e490f2f935a2-kube-api-access-kdlnz\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:03:47.032057 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.032021    2561 generic.go:356] "Generic (PLEG): container finished" podID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" containerID="342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256" exitCode=0
Jul 06 19:03:47.032576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.032073    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"
Jul 06 19:03:47.032576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.032083    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" event={"ID":"f0d6eba2-cd6d-48a4-8024-e490f2f935a2","Type":"ContainerDied","Data":"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"}
Jul 06 19:03:47.032576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.032127    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7" event={"ID":"f0d6eba2-cd6d-48a4-8024-e490f2f935a2","Type":"ContainerDied","Data":"66fed453ada6eebb91beb1f30810272d3f0aa83873568358b95d2019a18f64df"}
Jul 06 19:03:47.032576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.032151    2561 scope.go:117] "RemoveContainer" containerID="342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"
Jul 06 19:03:47.041346 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.041327    2561 scope.go:117] "RemoveContainer" containerID="342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"
Jul 06 19:03:47.041667 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:03:47.041647    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256\": container with ID starting with 342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256 not found: ID does not exist" containerID="342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"
Jul 06 19:03:47.041721 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.041677    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256"} err="failed to get container status \"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256\": rpc error: code = NotFound desc = could not find container \"342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256\": container with ID starting with 342c7b367c64fb8e4778fb93941c0f49e3e27114cd8577387a629ee84b301256 not found: ID does not exist"
Jul 06 19:03:47.058264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.058235    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:47.065550 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.065522    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jvjs7"]
Jul 06 19:03:47.961369 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:03:47.961330    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" path="/var/lib/kubelet/pods/f0d6eba2-cd6d-48a4-8024-e490f2f935a2/volumes"
Jul 06 19:04:04.435302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.435257    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:04.435708 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.435571    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" containerName="manager"
Jul 06 19:04:04.435708 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.435583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" containerName="manager"
Jul 06 19:04:04.435708 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.435634    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d6eba2-cd6d-48a4-8024-e490f2f935a2" containerName="manager"
Jul 06 19:04:04.476369 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.476336    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:04.476578 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.476432    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.479001 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.478976    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 19:04:04.525992 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.525959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.525992 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.525999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tbt\" (UniqueName: \"kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.532437 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.532409    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:04.627041 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.627004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.627255 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.627048    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tbt\" (UniqueName: \"kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.627763 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.627739    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.639199 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.639167    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tbt\" (UniqueName: \"kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt\") pod \"limitador-limitador-64c94fd64f-k92jg\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.787106 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.787014    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:04.955138 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:04.955098    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:04.958379 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:04:04.958346    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bab6de2_b883_4f41_9768_cef40fc1dcc1.slice/crio-546668e5dd7a092648a446fddbd8282d7325de121f188d4b32656e908a6dff1a WatchSource:0}: Error finding container 546668e5dd7a092648a446fddbd8282d7325de121f188d4b32656e908a6dff1a: Status 404 returned error can't find the container with id 546668e5dd7a092648a446fddbd8282d7325de121f188d4b32656e908a6dff1a
Jul 06 19:04:05.109797 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.109763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" event={"ID":"7bab6de2-b883-4f41-9768-cef40fc1dcc1","Type":"ContainerStarted","Data":"546668e5dd7a092648a446fddbd8282d7325de121f188d4b32656e908a6dff1a"}
Jul 06 19:04:05.635113 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.635075    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:05.639533 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.639513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:05.642087 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.642062    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b9zxp\""
Jul 06 19:04:05.644566 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.644541    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:05.736148 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.736116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfft\" (UniqueName: \"kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft\") pod \"authorino-75d556bd77-h4sfv\" (UID: \"f074ebf4-dd1c-485d-9f6c-cbe23b11120d\") " pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:05.837598 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.837555    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfft\" (UniqueName: \"kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft\") pod \"authorino-75d556bd77-h4sfv\" (UID: \"f074ebf4-dd1c-485d-9f6c-cbe23b11120d\") " pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:05.849995 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.849891    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfft\" (UniqueName: \"kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft\") pod \"authorino-75d556bd77-h4sfv\" (UID: \"f074ebf4-dd1c-485d-9f6c-cbe23b11120d\") " pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:05.950499 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:05.950404    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:06.158082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:06.158045    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:06.163069 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:04:06.163037    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf074ebf4_dd1c_485d_9f6c_cbe23b11120d.slice/crio-f01e4a8966f4f9d4c6400459d082f0e73001d1dd53616d37048edec12f4d419a WatchSource:0}: Error finding container f01e4a8966f4f9d4c6400459d082f0e73001d1dd53616d37048edec12f4d419a: Status 404 returned error can't find the container with id f01e4a8966f4f9d4c6400459d082f0e73001d1dd53616d37048edec12f4d419a
Jul 06 19:04:07.121378 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:07.121343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h4sfv" event={"ID":"f074ebf4-dd1c-485d-9f6c-cbe23b11120d","Type":"ContainerStarted","Data":"f01e4a8966f4f9d4c6400459d082f0e73001d1dd53616d37048edec12f4d419a"}
Jul 06 19:04:08.127285 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:08.127242    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" event={"ID":"7bab6de2-b883-4f41-9768-cef40fc1dcc1","Type":"ContainerStarted","Data":"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"}
Jul 06 19:04:08.127711 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:08.127379    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:08.157042 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:08.156976    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" podStartSLOduration=1.174293773 podStartE2EDuration="4.156952168s" totalImagesPullingTime="2.982658395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:04:08.15485997 +0000 UTC m=+550.741067945" watchObservedRunningTime="2026-07-06 19:04:08.156952168 +0000 UTC m=+550.743160142"
Jul 06 19:04:11.143096 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:11.143053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h4sfv" event={"ID":"f074ebf4-dd1c-485d-9f6c-cbe23b11120d","Type":"ContainerStarted","Data":"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"}
Jul 06 19:04:11.169056 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:11.169006    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-h4sfv" podStartSLOduration=2.068433631 podStartE2EDuration="6.168989444s" totalImagesPullingTime="4.100555813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:04:11.168026716 +0000 UTC m=+553.754234687" watchObservedRunningTime="2026-07-06 19:04:11.168989444 +0000 UTC m=+553.755197417"
Jul 06 19:04:19.133480 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:19.133440    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:19.508591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:19.508502    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:19.508761 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:19.508688    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" podUID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" containerName="limitador" containerID="cri-o://c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f" gracePeriod=30
Jul 06 19:04:20.070034 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.069965    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:20.163869 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.163832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6tbt\" (UniqueName: \"kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt\") pod \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") "
Jul 06 19:04:20.164363 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.163914    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file\") pod \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\" (UID: \"7bab6de2-b883-4f41-9768-cef40fc1dcc1\") "
Jul 06 19:04:20.164363 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.164264    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file" (OuterVolumeSpecName: "config-file") pod "7bab6de2-b883-4f41-9768-cef40fc1dcc1" (UID: "7bab6de2-b883-4f41-9768-cef40fc1dcc1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:04:20.166196 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.166170    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt" (OuterVolumeSpecName: "kube-api-access-t6tbt") pod "7bab6de2-b883-4f41-9768-cef40fc1dcc1" (UID: "7bab6de2-b883-4f41-9768-cef40fc1dcc1"). InnerVolumeSpecName "kube-api-access-t6tbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:04:20.181326 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.181292    2561 generic.go:356] "Generic (PLEG): container finished" podID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" containerID="c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f" exitCode=0
Jul 06 19:04:20.181531 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.181332    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" event={"ID":"7bab6de2-b883-4f41-9768-cef40fc1dcc1","Type":"ContainerDied","Data":"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"}
Jul 06 19:04:20.181531 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.181340    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg"
Jul 06 19:04:20.181531 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.181387    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k92jg" event={"ID":"7bab6de2-b883-4f41-9768-cef40fc1dcc1","Type":"ContainerDied","Data":"546668e5dd7a092648a446fddbd8282d7325de121f188d4b32656e908a6dff1a"}
Jul 06 19:04:20.181531 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.181417    2561 scope.go:117] "RemoveContainer" containerID="c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"
Jul 06 19:04:20.190761 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.190739    2561 scope.go:117] "RemoveContainer" containerID="c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"
Jul 06 19:04:20.191101 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:04:20.191083    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f\": container with ID starting with c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f not found: ID does not exist" containerID="c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"
Jul 06 19:04:20.191149 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.191111    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f"} err="failed to get container status \"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f\": rpc error: code = NotFound desc = could not find container \"c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f\": container with ID starting with c6834ca4af232837d0abef4a002b643ddfbe64413d0de21ff0fe05595b6b884f not found: ID does not exist"
Jul 06 19:04:20.202942 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.202910    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:20.211789 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.209817    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k92jg"]
Jul 06 19:04:20.264698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.264657    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7bab6de2-b883-4f41-9768-cef40fc1dcc1-config-file\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:04:20.264698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.264692    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6tbt\" (UniqueName: \"kubernetes.io/projected/7bab6de2-b883-4f41-9768-cef40fc1dcc1-kube-api-access-t6tbt\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:04:20.902694 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.902661    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-gbnsl"]
Jul 06 19:04:20.903002 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.902989    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" containerName="limitador"
Jul 06 19:04:20.903051 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.903003    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" containerName="limitador"
Jul 06 19:04:20.903082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.903050    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" containerName="limitador"
Jul 06 19:04:20.905844 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.905827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:20.908320 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.908298    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 19:04:20.908477 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.908338    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h8gcc\""
Jul 06 19:04:20.914650 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.914611    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-gbnsl"]
Jul 06 19:04:20.970582 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.970508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-data\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:20.970762 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:20.970639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2m4\" (UniqueName: \"kubernetes.io/projected/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-kube-api-access-nj2m4\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.071659 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.071617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-data\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.071836 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.071676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2m4\" (UniqueName: \"kubernetes.io/projected/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-kube-api-access-nj2m4\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.072014 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.071994    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-data\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.080571 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.080536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2m4\" (UniqueName: \"kubernetes.io/projected/a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed-kube-api-access-nj2m4\") pod \"postgres-69fb4594cf-gbnsl\" (UID: \"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed\") " pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.217362 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.217257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:21.352725 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.352695    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-gbnsl"]
Jul 06 19:04:21.355395 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:04:21.355365    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f90f3d_2299_46fa_bc93_e9f6ee4aa1ed.slice/crio-e91e25eda1c79ea485abdbd26778c97f9ac3d108dc22501d4eb20ca09c8d0b84 WatchSource:0}: Error finding container e91e25eda1c79ea485abdbd26778c97f9ac3d108dc22501d4eb20ca09c8d0b84: Status 404 returned error can't find the container with id e91e25eda1c79ea485abdbd26778c97f9ac3d108dc22501d4eb20ca09c8d0b84
Jul 06 19:04:21.961753 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:21.961717    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bab6de2-b883-4f41-9768-cef40fc1dcc1" path="/var/lib/kubelet/pods/7bab6de2-b883-4f41-9768-cef40fc1dcc1/volumes"
Jul 06 19:04:22.191919 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:22.191887    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-gbnsl" event={"ID":"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed","Type":"ContainerStarted","Data":"e91e25eda1c79ea485abdbd26778c97f9ac3d108dc22501d4eb20ca09c8d0b84"}
Jul 06 19:04:27.214771 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:27.214730    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-gbnsl" event={"ID":"a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed","Type":"ContainerStarted","Data":"9053ed31d96a8f0fea0e34f74c4d025145693c8e205f644ffad7204aac1281b0"}
Jul 06 19:04:27.215271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:27.214815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:27.229203 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:27.229146    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-gbnsl" podStartSLOduration=2.289529633 podStartE2EDuration="7.229127476s" totalImagesPullingTime="4.939597843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:04:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:04:27.229120608 +0000 UTC m=+569.815328582" watchObservedRunningTime="2026-07-06 19:04:27.229127476 +0000 UTC m=+569.815335451"
Jul 06 19:04:33.249711 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:33.249678    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-gbnsl"
Jul 06 19:04:35.631137 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:35.631097    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:35.631705 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:35.631369    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-h4sfv" podUID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" containerName="authorino" containerID="cri-o://ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d" gracePeriod=30
Jul 06 19:04:35.895479 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:35.895403    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:36.003088 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.003052    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcfft\" (UniqueName: \"kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft\") pod \"f074ebf4-dd1c-485d-9f6c-cbe23b11120d\" (UID: \"f074ebf4-dd1c-485d-9f6c-cbe23b11120d\") "
Jul 06 19:04:36.005278 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.005241    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft" (OuterVolumeSpecName: "kube-api-access-zcfft") pod "f074ebf4-dd1c-485d-9f6c-cbe23b11120d" (UID: "f074ebf4-dd1c-485d-9f6c-cbe23b11120d"). InnerVolumeSpecName "kube-api-access-zcfft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:04:36.104157 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.104109    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfft\" (UniqueName: \"kubernetes.io/projected/f074ebf4-dd1c-485d-9f6c-cbe23b11120d-kube-api-access-zcfft\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:04:36.252417 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.252318    2561 generic.go:356] "Generic (PLEG): container finished" podID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" containerID="ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d" exitCode=0
Jul 06 19:04:36.252417 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.252366    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-h4sfv"
Jul 06 19:04:36.252417 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.252378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h4sfv" event={"ID":"f074ebf4-dd1c-485d-9f6c-cbe23b11120d","Type":"ContainerDied","Data":"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"}
Jul 06 19:04:36.252718 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.252418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-h4sfv" event={"ID":"f074ebf4-dd1c-485d-9f6c-cbe23b11120d","Type":"ContainerDied","Data":"f01e4a8966f4f9d4c6400459d082f0e73001d1dd53616d37048edec12f4d419a"}
Jul 06 19:04:36.252718 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.252441    2561 scope.go:117] "RemoveContainer" containerID="ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"
Jul 06 19:04:36.261730 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.261709    2561 scope.go:117] "RemoveContainer" containerID="ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"
Jul 06 19:04:36.262034 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:04:36.262012    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d\": container with ID starting with ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d not found: ID does not exist" containerID="ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"
Jul 06 19:04:36.262076 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.262044    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d"} err="failed to get container status \"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d\": rpc error: code = NotFound desc = could not find container \"ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d\": container with ID starting with ab45465ebf69764ce6a07f2a6b2a1abc01a4a9a4cb179c258359b7901d10713d not found: ID does not exist"
Jul 06 19:04:36.282753 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.282723    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:36.289701 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:36.289675    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-h4sfv"]
Jul 06 19:04:37.961858 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:37.961828    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" path="/var/lib/kubelet/pods/f074ebf4-dd1c-485d-9f6c-cbe23b11120d/volumes"
Jul 06 19:04:39.940927 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.940890    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:39.941389 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.941208    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" containerName="authorino"
Jul 06 19:04:39.941389 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.941219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" containerName="authorino"
Jul 06 19:04:39.941389 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.941271    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f074ebf4-dd1c-485d-9f6c-cbe23b11120d" containerName="authorino"
Jul 06 19:04:39.992002 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.991973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:39.995234 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.995206    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kddn4\""
Jul 06 19:04:39.995515 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.995492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 19:04:39.995812 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.995795    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 19:04:39.998316 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:39.998294    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:40.137699 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.137668    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:04:40.140292 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.140266    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.140376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.140312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf2zw\" (UniqueName: \"kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.157106 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.157074    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:04:40.157305 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.157212    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.241049 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.240953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.241049 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.241004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf2zw\" (UniqueName: \"kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.243662 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.243636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.259614 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.259578    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf2zw\" (UniqueName: \"kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw\") pod \"maas-controller-748d58659f-kh2qm\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") " pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.302694 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.302658    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:40.341936 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.341906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.342065 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.341958    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568nw\" (UniqueName: \"kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.443080 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.443043    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568nw\" (UniqueName: \"kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.443268 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.443131    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.446034 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.446007    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.459151 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.459123    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568nw\" (UniqueName: \"kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw\") pod \"maas-controller-7bf8d74cb9-nvrjp\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") " pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.469161 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.468938    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:40.489790 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.489758    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:40.492785 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:04:40.492722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370131b5_30a5_4fbf_9232_484620918b15.slice/crio-cc47c2b0dc81fa9d2fd0d8e00f42fb629a650e04c948f31e5f8c534cc6b65481 WatchSource:0}: Error finding container cc47c2b0dc81fa9d2fd0d8e00f42fb629a650e04c948f31e5f8c534cc6b65481: Status 404 returned error can't find the container with id cc47c2b0dc81fa9d2fd0d8e00f42fb629a650e04c948f31e5f8c534cc6b65481
Jul 06 19:04:40.625257 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:40.625221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:04:40.628265 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:04:40.628222    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc27428_2a74_42d2_a47c_347c7a63ba6f.slice/crio-f574163a84c7d7d3cda039561f05b69e22ce4afbb1382c9d2e14f55d9e7a4550 WatchSource:0}: Error finding container f574163a84c7d7d3cda039561f05b69e22ce4afbb1382c9d2e14f55d9e7a4550: Status 404 returned error can't find the container with id f574163a84c7d7d3cda039561f05b69e22ce4afbb1382c9d2e14f55d9e7a4550
Jul 06 19:04:41.276967 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:41.276920    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" event={"ID":"2cc27428-2a74-42d2-a47c-347c7a63ba6f","Type":"ContainerStarted","Data":"f574163a84c7d7d3cda039561f05b69e22ce4afbb1382c9d2e14f55d9e7a4550"}
Jul 06 19:04:41.278272 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:41.278246    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-748d58659f-kh2qm" event={"ID":"370131b5-30a5-4fbf-9232-484620918b15","Type":"ContainerStarted","Data":"cc47c2b0dc81fa9d2fd0d8e00f42fb629a650e04c948f31e5f8c534cc6b65481"}
Jul 06 19:04:44.294201 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.294161    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-748d58659f-kh2qm" event={"ID":"370131b5-30a5-4fbf-9232-484620918b15","Type":"ContainerStarted","Data":"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"}
Jul 06 19:04:44.294702 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.294331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:44.295662 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.295633    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" event={"ID":"2cc27428-2a74-42d2-a47c-347c7a63ba6f","Type":"ContainerStarted","Data":"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"}
Jul 06 19:04:44.295745 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.295698    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:44.348129 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.348075    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-748d58659f-kh2qm" podStartSLOduration=2.231363692 podStartE2EDuration="5.348056461s" totalImagesPullingTime="3.116692769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:04:44.346870989 +0000 UTC m=+586.933078981" watchObservedRunningTime="2026-07-06 19:04:44.348056461 +0000 UTC m=+586.934264434"
Jul 06 19:04:44.396858 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:44.396798    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" podStartSLOduration=1.424433907 podStartE2EDuration="4.39677876s" totalImagesPullingTime="2.972344853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:04:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:04:44.39153076 +0000 UTC m=+586.977738734" watchObservedRunningTime="2026-07-06 19:04:44.39677876 +0000 UTC m=+586.982986733"
Jul 06 19:04:55.306339 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.306304    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:55.308053 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.308030    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:04:55.398018 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.397983    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:55.398247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.398207    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-748d58659f-kh2qm" podUID="370131b5-30a5-4fbf-9232-484620918b15" containerName="manager" containerID="cri-o://bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2" gracePeriod=30
Jul 06 19:04:55.640257 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.640227    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:55.664330 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.664297    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert\") pod \"370131b5-30a5-4fbf-9232-484620918b15\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") "
Jul 06 19:04:55.664538 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.664347    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf2zw\" (UniqueName: \"kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw\") pod \"370131b5-30a5-4fbf-9232-484620918b15\" (UID: \"370131b5-30a5-4fbf-9232-484620918b15\") "
Jul 06 19:04:55.666977 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.666936    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "370131b5-30a5-4fbf-9232-484620918b15" (UID: "370131b5-30a5-4fbf-9232-484620918b15"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:04:55.667097 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.666979    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw" (OuterVolumeSpecName: "kube-api-access-sf2zw") pod "370131b5-30a5-4fbf-9232-484620918b15" (UID: "370131b5-30a5-4fbf-9232-484620918b15"). InnerVolumeSpecName "kube-api-access-sf2zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:04:55.765562 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.765524    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/370131b5-30a5-4fbf-9232-484620918b15-webhook-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:04:55.765562 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:55.765554    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf2zw\" (UniqueName: \"kubernetes.io/projected/370131b5-30a5-4fbf-9232-484620918b15-kube-api-access-sf2zw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:04:56.357876 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.357840    2561 generic.go:356] "Generic (PLEG): container finished" podID="370131b5-30a5-4fbf-9232-484620918b15" containerID="bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2" exitCode=0
Jul 06 19:04:56.358271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.357892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-748d58659f-kh2qm"
Jul 06 19:04:56.358271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.357926    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-748d58659f-kh2qm" event={"ID":"370131b5-30a5-4fbf-9232-484620918b15","Type":"ContainerDied","Data":"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"}
Jul 06 19:04:56.358271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.357985    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-748d58659f-kh2qm" event={"ID":"370131b5-30a5-4fbf-9232-484620918b15","Type":"ContainerDied","Data":"cc47c2b0dc81fa9d2fd0d8e00f42fb629a650e04c948f31e5f8c534cc6b65481"}
Jul 06 19:04:56.358271 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.358016    2561 scope.go:117] "RemoveContainer" containerID="bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"
Jul 06 19:04:56.366819 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.366799    2561 scope.go:117] "RemoveContainer" containerID="bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"
Jul 06 19:04:56.367109 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:04:56.367090    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2\": container with ID starting with bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2 not found: ID does not exist" containerID="bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"
Jul 06 19:04:56.367156 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.367125    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2"} err="failed to get container status \"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2\": rpc error: code = NotFound desc = could not find container \"bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2\": container with ID starting with bf649e04169f0f143905290f1cfedba2c5193d95ea23d448554fd0fc2cff5cc2 not found: ID does not exist"
Jul 06 19:04:56.383414 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.383377    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:56.385791 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:56.385764    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-748d58659f-kh2qm"]
Jul 06 19:04:57.967986 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:04:57.967933    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370131b5-30a5-4fbf-9232-484620918b15" path="/var/lib/kubelet/pods/370131b5-30a5-4fbf-9232-484620918b15/volumes"
Jul 06 19:05:16.133950 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.133916    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-68cb9b94f4-xzgkp"]
Jul 06 19:05:16.134602 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.134366    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370131b5-30a5-4fbf-9232-484620918b15" containerName="manager"
Jul 06 19:05:16.134602 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.134385    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="370131b5-30a5-4fbf-9232-484620918b15" containerName="manager"
Jul 06 19:05:16.134602 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.134507    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="370131b5-30a5-4fbf-9232-484620918b15" containerName="manager"
Jul 06 19:05:16.140257 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.140231    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.143391 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.143368    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-6wq86\""
Jul 06 19:05:16.143933 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.143913    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
Jul 06 19:05:16.151856 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.151830    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68cb9b94f4-xzgkp"]
Jul 06 19:05:16.243061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.243004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s289l\" (UniqueName: \"kubernetes.io/projected/afda3049-3c2b-4af3-9800-e9688e7d73d9-kube-api-access-s289l\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.243254 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.243128    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afda3049-3c2b-4af3-9800-e9688e7d73d9-openshift-service-ca\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.243254 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.243170    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afda3049-3c2b-4af3-9800-e9688e7d73d9-maas-api-tls\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.344061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.344024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afda3049-3c2b-4af3-9800-e9688e7d73d9-openshift-service-ca\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.344061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.344063    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afda3049-3c2b-4af3-9800-e9688e7d73d9-maas-api-tls\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.344291 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.344108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s289l\" (UniqueName: \"kubernetes.io/projected/afda3049-3c2b-4af3-9800-e9688e7d73d9-kube-api-access-s289l\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.344767 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.344729    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afda3049-3c2b-4af3-9800-e9688e7d73d9-openshift-service-ca\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.346661 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.346642    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afda3049-3c2b-4af3-9800-e9688e7d73d9-maas-api-tls\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.353726 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.353693    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s289l\" (UniqueName: \"kubernetes.io/projected/afda3049-3c2b-4af3-9800-e9688e7d73d9-kube-api-access-s289l\") pod \"maas-api-68cb9b94f4-xzgkp\" (UID: \"afda3049-3c2b-4af3-9800-e9688e7d73d9\") " pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.451691 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.451603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:16.632207 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:16.632183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-68cb9b94f4-xzgkp"]
Jul 06 19:05:16.634390 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:05:16.634364    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafda3049_3c2b_4af3_9800_e9688e7d73d9.slice/crio-5d46b2dc2908de345d8f4a873aa5c15c9dfe784a2685837d2fb807334c582bc1 WatchSource:0}: Error finding container 5d46b2dc2908de345d8f4a873aa5c15c9dfe784a2685837d2fb807334c582bc1: Status 404 returned error can't find the container with id 5d46b2dc2908de345d8f4a873aa5c15c9dfe784a2685837d2fb807334c582bc1
Jul 06 19:05:17.002522 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.002484    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-thrsd"]
Jul 06 19:05:17.012286 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.012257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.015980 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.015956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 19:05:17.016231 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.016206    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 19:05:17.016437 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.016424    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 19:05:17.041890 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.041348    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-thrsd"]
Jul 06 19:05:17.150809 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.150767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-plugins-config-volume\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.151200 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.150869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf44h\" (UniqueName: \"kubernetes.io/projected/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-kube-api-access-zf44h\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.251543 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.251506    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf44h\" (UniqueName: \"kubernetes.io/projected/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-kube-api-access-zf44h\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.251744 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.251568    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-plugins-config-volume\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.252227 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.252207    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-plugins-config-volume\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.265187 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.265118    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf44h\" (UniqueName: \"kubernetes.io/projected/f5f894d5-6cd5-4bbf-99eb-82e4d9d91458-kube-api-access-zf44h\") pod \"payload-pre-processing-7964457696-thrsd\" (UID: \"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458\") " pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.323058 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.323019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:17.454802 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.454767    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68cb9b94f4-xzgkp" event={"ID":"afda3049-3c2b-4af3-9800-e9688e7d73d9","Type":"ContainerStarted","Data":"5d46b2dc2908de345d8f4a873aa5c15c9dfe784a2685837d2fb807334c582bc1"}
Jul 06 19:05:17.501578 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:17.501540    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-thrsd"]
Jul 06 19:05:17.502877 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:05:17.502844    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f894d5_6cd5_4bbf_99eb_82e4d9d91458.slice/crio-42f11a359a069972a061319626bd6a6078809853591b2904a69bf1f2aeab2b6e WatchSource:0}: Error finding container 42f11a359a069972a061319626bd6a6078809853591b2904a69bf1f2aeab2b6e: Status 404 returned error can't find the container with id 42f11a359a069972a061319626bd6a6078809853591b2904a69bf1f2aeab2b6e
Jul 06 19:05:18.462579 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:18.462450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-68cb9b94f4-xzgkp" event={"ID":"afda3049-3c2b-4af3-9800-e9688e7d73d9","Type":"ContainerStarted","Data":"c9afde008975a338460bcadcb4a9809b2bc2796b500a1f44cf4c3deb9188a29b"}
Jul 06 19:05:18.462977 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:18.462670    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:18.464058 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:18.464029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd" event={"ID":"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458","Type":"ContainerStarted","Data":"42f11a359a069972a061319626bd6a6078809853591b2904a69bf1f2aeab2b6e"}
Jul 06 19:05:18.484113 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:18.484057    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-68cb9b94f4-xzgkp" podStartSLOduration=0.978756357 podStartE2EDuration="2.484033849s" totalImagesPullingTime="1.505277492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:18.482393492 +0000 UTC m=+621.068601483" watchObservedRunningTime="2026-07-06 19:05:18.484033849 +0000 UTC m=+621.070241824"
Jul 06 19:05:19.469815 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:19.469714    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd" event={"ID":"f5f894d5-6cd5-4bbf-99eb-82e4d9d91458","Type":"ContainerStarted","Data":"7b42e85256c98fb5fc6a0f162dba6c59c0cb15dbdff07336ddd9529e7f1b22be"}
Jul 06 19:05:19.470190 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:19.469851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:19.498306 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:19.498244    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd" podStartSLOduration=1.788027534 podStartE2EDuration="3.498227791s" totalImagesPullingTime="1.710200257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:19.496287272 +0000 UTC m=+622.082495264" watchObservedRunningTime="2026-07-06 19:05:19.498227791 +0000 UTC m=+622.084435764"
Jul 06 19:05:24.475203 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:24.475175    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-68cb9b94f4-xzgkp"
Jul 06 19:05:30.477304 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:30.477273    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-thrsd"
Jul 06 19:05:31.825455 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.825419    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"]
Jul 06 19:05:31.829061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.829034    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.833531 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.833495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 19:05:31.833828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.833805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 06 19:05:31.833916 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.833846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-crqlx\""
Jul 06 19:05:31.833916 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.833875    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 19:05:31.857431 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.857395    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"]
Jul 06 19:05:31.974189 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974150    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.974376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.974376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.974376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.974376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:31.974376 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:31.974368    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p92nr\" (UniqueName: \"kubernetes.io/projected/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-kube-api-access-p92nr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.075677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075580    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p92nr\" (UniqueName: \"kubernetes.io/projected/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-kube-api-access-p92nr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.075677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075634    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.075677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076065 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076065 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076065 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.075771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076161 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.076141    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076198 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.076180    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.076283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.076259    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.078244 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.078219    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.078491 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.078451    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.085009 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.084987    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p92nr\" (UniqueName: \"kubernetes.io/projected/43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b-kube-api-access-p92nr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc\" (UID: \"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.139204 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.139167    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:32.287008 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.286960    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"]
Jul 06 19:05:32.293757 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:05:32.293728    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d79110_14d4_4e6d_bb8f_ff1ffeb7e76b.slice/crio-a50b6abc0cf1d5b7f9ca43d97b0254c5d001c0f6c5a1399c77c31c9690639287 WatchSource:0}: Error finding container a50b6abc0cf1d5b7f9ca43d97b0254c5d001c0f6c5a1399c77c31c9690639287: Status 404 returned error can't find the container with id a50b6abc0cf1d5b7f9ca43d97b0254c5d001c0f6c5a1399c77c31c9690639287
Jul 06 19:05:32.526535 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:32.526418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc" event={"ID":"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b","Type":"ContainerStarted","Data":"a50b6abc0cf1d5b7f9ca43d97b0254c5d001c0f6c5a1399c77c31c9690639287"}
Jul 06 19:05:34.537792 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:34.537753    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc" event={"ID":"43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b","Type":"ContainerStarted","Data":"4ee4baf7e4a0d6a54586e1b0350b27cecaf4d56e65b327d697b466b0db9f61b4"}
Jul 06 19:05:34.561473 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:34.561416    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc" podStartSLOduration=2.342776971 podStartE2EDuration="3.561395142s" totalImagesPullingTime="1.218618171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:34.559473904 +0000 UTC m=+637.145681879" watchObservedRunningTime="2026-07-06 19:05:34.561395142 +0000 UTC m=+637.147603117"
Jul 06 19:05:36.291880 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.291843    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"]
Jul 06 19:05:36.294884 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.294866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.297383 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.297357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 06 19:05:36.305782 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.305753    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"]
Jul 06 19:05:36.416661 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.416661 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.416931 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.416931 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.416931 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lps5j\" (UniqueName: \"kubernetes.io/projected/aadce368-26e4-4c19-895d-b4c05237d34b-kube-api-access-lps5j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.416931 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.416895    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadce368-26e4-4c19-895d-b4c05237d34b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.517844 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.517784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.517844 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.517848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lps5j\" (UniqueName: \"kubernetes.io/projected/aadce368-26e4-4c19-895d-b4c05237d34b-kube-api-access-lps5j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.517871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadce368-26e4-4c19-895d-b4c05237d34b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.517921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.517980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.518034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518296 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.518237    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518440 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.518407    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.518530 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.518446    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.520433 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.520409    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadce368-26e4-4c19-895d-b4c05237d34b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.520698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.520683    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadce368-26e4-4c19-895d-b4c05237d34b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.527897 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.527870    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lps5j\" (UniqueName: \"kubernetes.io/projected/aadce368-26e4-4c19-895d-b4c05237d34b-kube-api-access-lps5j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k94d2\" (UID: \"aadce368-26e4-4c19-895d-b4c05237d34b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.606391 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.606346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:36.754165 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:36.754131    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"]
Jul 06 19:05:36.755447 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:05:36.755421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadce368_26e4_4c19_895d_b4c05237d34b.slice/crio-781b117d85ed6d75e00973cf08e6e5e19b048905fa33efafc3de1c89eaab860e WatchSource:0}: Error finding container 781b117d85ed6d75e00973cf08e6e5e19b048905fa33efafc3de1c89eaab860e: Status 404 returned error can't find the container with id 781b117d85ed6d75e00973cf08e6e5e19b048905fa33efafc3de1c89eaab860e
Jul 06 19:05:37.554087 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:37.554054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2" event={"ID":"aadce368-26e4-4c19-895d-b4c05237d34b","Type":"ContainerStarted","Data":"b02c4836fc6488a0d1682daa850be1a593c90ad5b63f06e1c7605a3dede9ab6e"}
Jul 06 19:05:37.554671 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:37.554103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2" event={"ID":"aadce368-26e4-4c19-895d-b4c05237d34b","Type":"ContainerStarted","Data":"781b117d85ed6d75e00973cf08e6e5e19b048905fa33efafc3de1c89eaab860e"}
Jul 06 19:05:37.577218 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:37.577156    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2" podStartSLOduration=1.391305135 podStartE2EDuration="1.577137496s" totalImagesPullingTime="185.832361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:37.575694091 +0000 UTC m=+640.161902062" watchObservedRunningTime="2026-07-06 19:05:37.577137496 +0000 UTC m=+640.163345470"
Jul 06 19:05:42.139899 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:42.139860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:42.140293 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:42.139940    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:42.153018 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:42.152986    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:42.588691 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:42.588660    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc"
Jul 06 19:05:46.606754 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:46.606721    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:46.607194 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:46.606990    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:46.619653 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:46.619620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:05:47.610169 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:05:47.610133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k94d2"
Jul 06 19:06:04.326761 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.326729    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"]
Jul 06 19:06:04.330176 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.330154    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.332644 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.332623    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 06 19:06:04.342513 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.342487    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"]
Jul 06 19:06:04.454846 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.454803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.454846 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.454848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.455082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.454882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.455082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.454936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.455082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.454982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnbxj\" (UniqueName: \"kubernetes.io/projected/be7d83fc-0f48-497f-856e-c3f95d2bfb73-kube-api-access-pnbxj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.455082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.455021    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556015 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.555977    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556015 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556272 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556272 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556272 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnbxj\" (UniqueName: \"kubernetes.io/projected/be7d83fc-0f48-497f-856e-c3f95d2bfb73-kube-api-access-pnbxj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556272 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556596 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556562    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556724 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556602    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.556724 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.556572    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.559056 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.559034    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be7d83fc-0f48-497f-856e-c3f95d2bfb73-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.559233 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.559215    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be7d83fc-0f48-497f-856e-c3f95d2bfb73-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.564139 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.564116    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnbxj\" (UniqueName: \"kubernetes.io/projected/be7d83fc-0f48-497f-856e-c3f95d2bfb73-kube-api-access-pnbxj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5\" (UID: \"be7d83fc-0f48-497f-856e-c3f95d2bfb73\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.640224 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.640131    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:04.786312 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.786221    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"]
Jul 06 19:06:04.790194 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:06:04.790166    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7d83fc_0f48_497f_856e_c3f95d2bfb73.slice/crio-e063def856b6a465fa889887a903fa4c4803270489cd747d4f81e4ca5a1b200f WatchSource:0}: Error finding container e063def856b6a465fa889887a903fa4c4803270489cd747d4f81e4ca5a1b200f: Status 404 returned error can't find the container with id e063def856b6a465fa889887a903fa4c4803270489cd747d4f81e4ca5a1b200f
Jul 06 19:06:04.791712 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:04.791692    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:06:05.682786 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:05.682747    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5" event={"ID":"be7d83fc-0f48-497f-856e-c3f95d2bfb73","Type":"ContainerStarted","Data":"dfe9bc7dfe356ca8fec212c03e52d7d45a14783e3a35b0b23d4f1c1b0787b39c"}
Jul 06 19:06:05.683182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:05.682811    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5" event={"ID":"be7d83fc-0f48-497f-856e-c3f95d2bfb73","Type":"ContainerStarted","Data":"e063def856b6a465fa889887a903fa4c4803270489cd747d4f81e4ca5a1b200f"}
Jul 06 19:06:05.704037 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:05.703979    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5" podStartSLOduration=1.462125827 podStartE2EDuration="1.703960273s" totalImagesPullingTime="241.834446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:05.702046485 +0000 UTC m=+668.288254459" watchObservedRunningTime="2026-07-06 19:06:05.703960273 +0000 UTC m=+668.290168247"
Jul 06 19:06:14.641233 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:14.641191    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:14.641755 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:14.641720    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:14.659740 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:14.659710    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:06:14.734490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:06:14.734429    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5"
Jul 06 19:09:05.458322 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.458286    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:09:05.458917 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.458508    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" podUID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" containerName="manager" containerID="cri-o://c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c" gracePeriod=30
Jul 06 19:09:05.712796 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.712728    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:09:05.806715 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.806677    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-568nw\" (UniqueName: \"kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw\") pod \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") "
Jul 06 19:09:05.806883 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.806728    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert\") pod \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\" (UID: \"2cc27428-2a74-42d2-a47c-347c7a63ba6f\") "
Jul 06 19:09:05.808919 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.808889    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw" (OuterVolumeSpecName: "kube-api-access-568nw") pod "2cc27428-2a74-42d2-a47c-347c7a63ba6f" (UID: "2cc27428-2a74-42d2-a47c-347c7a63ba6f"). InnerVolumeSpecName "kube-api-access-568nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:09:05.808997 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.808910    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2cc27428-2a74-42d2-a47c-347c7a63ba6f" (UID: "2cc27428-2a74-42d2-a47c-347c7a63ba6f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:09:05.908153 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.908110    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568nw\" (UniqueName: \"kubernetes.io/projected/2cc27428-2a74-42d2-a47c-347c7a63ba6f-kube-api-access-568nw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:09:05.908153 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:05.908145    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc27428-2a74-42d2-a47c-347c7a63ba6f-webhook-cert\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:09:06.479725 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.479687    2561 generic.go:356] "Generic (PLEG): container finished" podID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" containerID="c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c" exitCode=0
Jul 06 19:09:06.480231 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.479753    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp"
Jul 06 19:09:06.480231 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.479775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" event={"ID":"2cc27428-2a74-42d2-a47c-347c7a63ba6f","Type":"ContainerDied","Data":"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"}
Jul 06 19:09:06.480231 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.479831    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-nvrjp" event={"ID":"2cc27428-2a74-42d2-a47c-347c7a63ba6f","Type":"ContainerDied","Data":"f574163a84c7d7d3cda039561f05b69e22ce4afbb1382c9d2e14f55d9e7a4550"}
Jul 06 19:09:06.480231 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.479853    2561 scope.go:117] "RemoveContainer" containerID="c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"
Jul 06 19:09:06.488386 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.488366    2561 scope.go:117] "RemoveContainer" containerID="c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"
Jul 06 19:09:06.488687 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:09:06.488664    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c\": container with ID starting with c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c not found: ID does not exist" containerID="c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"
Jul 06 19:09:06.488762 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.488695    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c"} err="failed to get container status \"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c\": rpc error: code = NotFound desc = could not find container \"c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c\": container with ID starting with c95acd6f5699873069caee6c0d575d171696159d07e1100eb0dcfe953adf7a7c not found: ID does not exist"
Jul 06 19:09:06.504057 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.504029    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:09:06.508233 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:06.508207    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-nvrjp"]
Jul 06 19:09:07.267987 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.267945    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-tfnq8"]
Jul 06 19:09:07.268346 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.268329    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" containerName="manager"
Jul 06 19:09:07.268436 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.268350    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" containerName="manager"
Jul 06 19:09:07.268513 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.268481    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" containerName="manager"
Jul 06 19:09:07.272690 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.272668    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.275476 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.275438    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 06 19:09:07.275624 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.275492    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 06 19:09:07.275624 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.275438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kddn4\""
Jul 06 19:09:07.281393 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.281370    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-tfnq8"]
Jul 06 19:09:07.321602 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.321539    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg5dw\" (UniqueName: \"kubernetes.io/projected/a6aab718-837a-4515-80a6-7379f5b5b0f8-kube-api-access-hg5dw\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.321801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.321635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6aab718-837a-4515-80a6-7379f5b5b0f8-webhook-cert\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.422801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.422755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg5dw\" (UniqueName: \"kubernetes.io/projected/a6aab718-837a-4515-80a6-7379f5b5b0f8-kube-api-access-hg5dw\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.422801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.422812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6aab718-837a-4515-80a6-7379f5b5b0f8-webhook-cert\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.425573 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.425545    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a6aab718-837a-4515-80a6-7379f5b5b0f8-webhook-cert\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.431965 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.431929    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg5dw\" (UniqueName: \"kubernetes.io/projected/a6aab718-837a-4515-80a6-7379f5b5b0f8-kube-api-access-hg5dw\") pod \"maas-controller-7bf8d74cb9-tfnq8\" (UID: \"a6aab718-837a-4515-80a6-7379f5b5b0f8\") " pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.584056 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.584019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:07.724416 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.724392    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bf8d74cb9-tfnq8"]
Jul 06 19:09:07.727185 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:09:07.727156    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aab718_837a_4515_80a6_7379f5b5b0f8.slice/crio-297fb5098f025c31c6a5a98fa55657c9adda23bc7d773fce14aa0b9915e0237f WatchSource:0}: Error finding container 297fb5098f025c31c6a5a98fa55657c9adda23bc7d773fce14aa0b9915e0237f: Status 404 returned error can't find the container with id 297fb5098f025c31c6a5a98fa55657c9adda23bc7d773fce14aa0b9915e0237f
Jul 06 19:09:07.961943 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:07.961809    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cc27428-2a74-42d2-a47c-347c7a63ba6f" path="/var/lib/kubelet/pods/2cc27428-2a74-42d2-a47c-347c7a63ba6f/volumes"
Jul 06 19:09:08.491944 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:08.491907    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" event={"ID":"a6aab718-837a-4515-80a6-7379f5b5b0f8","Type":"ContainerStarted","Data":"a003ac761bacbf2b4a5c34c0064da73d61004ad22a0953a1847543fb01cb654c"}
Jul 06 19:09:08.492118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:08.491958    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" event={"ID":"a6aab718-837a-4515-80a6-7379f5b5b0f8","Type":"ContainerStarted","Data":"297fb5098f025c31c6a5a98fa55657c9adda23bc7d773fce14aa0b9915e0237f"}
Jul 06 19:09:08.492118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:08.492040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:09:08.512844 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:08.512791    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" podStartSLOduration=0.952280127 podStartE2EDuration="1.512766975s" totalImagesPullingTime="560.486848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:08.511498133 +0000 UTC m=+851.097706107" watchObservedRunningTime="2026-07-06 19:09:08.512766975 +0000 UTC m=+851.098974949"
Jul 06 19:09:19.504612 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:09:19.504577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8"
Jul 06 19:14:01.313637 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:01.313605    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:14:01.316364 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:01.313806    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" podUID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" containerName="manager" containerID="cri-o://af8257ff083a6b2766d6973368777537c366b4a3e63c65ab58f66115b6c6a403" gracePeriod=10
Jul 06 19:14:04.788789 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:04.788753    2561 generic.go:356] "Generic (PLEG): container finished" podID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" containerID="af8257ff083a6b2766d6973368777537c366b4a3e63c65ab58f66115b6c6a403" exitCode=0
Jul 06 19:14:04.789180 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:04.788822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" event={"ID":"79fcfdc9-459e-4ce6-a1f4-657aaa73313b","Type":"ContainerDied","Data":"af8257ff083a6b2766d6973368777537c366b4a3e63c65ab58f66115b6c6a403"}
Jul 06 19:14:05.266324 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.266297    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:14:05.332744 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.332708    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n47vj\" (UniqueName: \"kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj\") pod \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") "
Jul 06 19:14:05.332899 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.332800    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume\") pod \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\" (UID: \"79fcfdc9-459e-4ce6-a1f4-657aaa73313b\") "
Jul 06 19:14:05.333167 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.333141    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "79fcfdc9-459e-4ce6-a1f4-657aaa73313b" (UID: "79fcfdc9-459e-4ce6-a1f4-657aaa73313b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:14:05.334996 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.334966    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj" (OuterVolumeSpecName: "kube-api-access-n47vj") pod "79fcfdc9-459e-4ce6-a1f4-657aaa73313b" (UID: "79fcfdc9-459e-4ce6-a1f4-657aaa73313b"). InnerVolumeSpecName "kube-api-access-n47vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:14:05.434516 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.434396    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47vj\" (UniqueName: \"kubernetes.io/projected/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-kube-api-access-n47vj\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:14:05.434516 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.434427    2561 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/79fcfdc9-459e-4ce6-a1f4-657aaa73313b-extensions-socket-volume\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:14:05.794566 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.794446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx" event={"ID":"79fcfdc9-459e-4ce6-a1f4-657aaa73313b","Type":"ContainerDied","Data":"85d8a72e5f22b4b9aba76dac7b35a2c626d6a75ade3abc316cf456c2872b7cf2"}
Jul 06 19:14:05.794566 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.794567    2561 scope.go:117] "RemoveContainer" containerID="af8257ff083a6b2766d6973368777537c366b4a3e63c65ab58f66115b6c6a403"
Jul 06 19:14:05.795072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.794485    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"
Jul 06 19:14:05.817293 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.817257    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:14:05.826199 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.826173    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4scfx"]
Jul 06 19:14:05.962327 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:14:05.962296    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" path="/var/lib/kubelet/pods/79fcfdc9-459e-4ce6-a1f4-657aaa73313b/volumes"
Jul 06 19:15:02.546222 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.546187    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"]
Jul 06 19:15:02.546668 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.546536    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" containerName="manager"
Jul 06 19:15:02.546668 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.546547    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" containerName="manager"
Jul 06 19:15:02.546668 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.546613    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="79fcfdc9-459e-4ce6-a1f4-657aaa73313b" containerName="manager"
Jul 06 19:15:02.550696 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.550677    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.553741 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.553720    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7rhkt\""
Jul 06 19:15:02.573634 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.573604    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"]
Jul 06 19:15:02.650754 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.650712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m54j\" (UniqueName: \"kubernetes.io/projected/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-kube-api-access-9m54j\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.650935 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.650830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.751949 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.751909    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.752084 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.751976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m54j\" (UniqueName: \"kubernetes.io/projected/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-kube-api-access-9m54j\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.752332 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.752313    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.766829 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.766792    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m54j\" (UniqueName: \"kubernetes.io/projected/527770fd-d5f9-4520-8fb4-bfbcbd28b70d-kube-api-access-9m54j\") pod \"kuadrant-operator-controller-manager-86878c769f-5wqw8\" (UID: \"527770fd-d5f9-4520-8fb4-bfbcbd28b70d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:02.861248 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:02.861205    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:03.017701 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:03.017625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"]
Jul 06 19:15:03.020390 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:15:03.020360    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527770fd_d5f9_4520_8fb4_bfbcbd28b70d.slice/crio-c409aa6c4f8a1dd64b3e88d71bf6682b153e1e1f67c2d90dbf4a49d7165ce36c WatchSource:0}: Error finding container c409aa6c4f8a1dd64b3e88d71bf6682b153e1e1f67c2d90dbf4a49d7165ce36c: Status 404 returned error can't find the container with id c409aa6c4f8a1dd64b3e88d71bf6682b153e1e1f67c2d90dbf4a49d7165ce36c
Jul 06 19:15:03.047053 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:03.047025    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8" event={"ID":"527770fd-d5f9-4520-8fb4-bfbcbd28b70d","Type":"ContainerStarted","Data":"c409aa6c4f8a1dd64b3e88d71bf6682b153e1e1f67c2d90dbf4a49d7165ce36c"}
Jul 06 19:15:04.053027 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:04.052988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8" event={"ID":"527770fd-d5f9-4520-8fb4-bfbcbd28b70d","Type":"ContainerStarted","Data":"92a91a0a26e25283c90c93f59f82fcba7fead4b49fe70e9d9fe7790d2fa72e94"}
Jul 06 19:15:04.053547 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:04.053123    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:15:04.083520 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:04.083446    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8" podStartSLOduration=2.083426248 podStartE2EDuration="2.083426248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:04.080939029 +0000 UTC m=+1206.667147005" watchObservedRunningTime="2026-07-06 19:15:04.083426248 +0000 UTC m=+1206.669634222"
Jul 06 19:15:15.059774 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:15:15.059743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wqw8"
Jul 06 19:21:26.339940 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.339907    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:26.343582 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.343559    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.346418 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.346395    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-1bd5e6c9-serving-cert\""
Jul 06 19:21:26.381237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.381190    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:26.399206 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.399164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.399382 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.399241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.399382 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.399294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kssnn\" (UniqueName: \"kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.500086 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.500040    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.500302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.500112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kssnn\" (UniqueName: \"kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.500302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.500145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.500302 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:26.500243    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-1bd5e6c9-serving-cert: secret "maas-api-e2e-ait-1bd5e6c9-serving-cert" not found
Jul 06 19:21:26.500476 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:26.500320    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls podName:856f7d39-9e42-4bbc-afaa-58ef30a34aac nodeName:}" failed. No retries permitted until 2026-07-06 19:21:27.00029557 +0000 UTC m=+1589.586503525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls") pod "maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" (UID: "856f7d39-9e42-4bbc-afaa-58ef30a34aac") : secret "maas-api-e2e-ait-1bd5e6c9-serving-cert" not found
Jul 06 19:21:26.500795 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.500774    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:26.521304 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:26.521270    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kssnn\" (UniqueName: \"kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:27.005446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.005408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:27.008044 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.008016    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") pod \"maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") " pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:27.254483 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.254423    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:27.411975 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.411944    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:27.415127 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:21:27.415098    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856f7d39_9e42_4bbc_afaa_58ef30a34aac.slice/crio-76be6f84df54150a435cb54c24d0cd3e0a32fe8267024ab3d8b728ade1a31fa6 WatchSource:0}: Error finding container 76be6f84df54150a435cb54c24d0cd3e0a32fe8267024ab3d8b728ade1a31fa6: Status 404 returned error can't find the container with id 76be6f84df54150a435cb54c24d0cd3e0a32fe8267024ab3d8b728ade1a31fa6
Jul 06 19:21:27.416500 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.416482    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:21:27.754031 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:27.753995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" event={"ID":"856f7d39-9e42-4bbc-afaa-58ef30a34aac","Type":"ContainerStarted","Data":"76be6f84df54150a435cb54c24d0cd3e0a32fe8267024ab3d8b728ade1a31fa6"}
Jul 06 19:21:28.763287 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:28.762963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" event={"ID":"856f7d39-9e42-4bbc-afaa-58ef30a34aac","Type":"ContainerStarted","Data":"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"}
Jul 06 19:21:28.763287 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:28.763098    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:28.796407 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:28.796340    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" podStartSLOduration=2.294309519 podStartE2EDuration="2.7963145s" totalImagesPullingTime="502.004981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:28.793791615 +0000 UTC m=+1591.379999620" watchObservedRunningTime="2026-07-06 19:21:28.7963145 +0000 UTC m=+1591.382522477"
Jul 06 19:21:31.732035 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:31.731997    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:31.732565 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:31.732196    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" podUID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" containerName="maas-api" containerID="cri-o://67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354" gracePeriod=30
Jul 06 19:21:31.988009 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:31.987945    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:32.053098 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.053059    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") pod \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") "
Jul 06 19:21:32.053098 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.053101    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca\") pod \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") "
Jul 06 19:21:32.053353 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.053183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kssnn\" (UniqueName: \"kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn\") pod \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\" (UID: \"856f7d39-9e42-4bbc-afaa-58ef30a34aac\") "
Jul 06 19:21:32.053601 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.053542    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "856f7d39-9e42-4bbc-afaa-58ef30a34aac" (UID: "856f7d39-9e42-4bbc-afaa-58ef30a34aac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:21:32.055527 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.055498    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "856f7d39-9e42-4bbc-afaa-58ef30a34aac" (UID: "856f7d39-9e42-4bbc-afaa-58ef30a34aac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:21:32.055640 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.055533    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn" (OuterVolumeSpecName: "kube-api-access-kssnn") pod "856f7d39-9e42-4bbc-afaa-58ef30a34aac" (UID: "856f7d39-9e42-4bbc-afaa-58ef30a34aac"). InnerVolumeSpecName "kube-api-access-kssnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:21:32.154028 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.153997    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/856f7d39-9e42-4bbc-afaa-58ef30a34aac-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:32.154028 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.154026    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/856f7d39-9e42-4bbc-afaa-58ef30a34aac-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:32.154028 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.154036    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kssnn\" (UniqueName: \"kubernetes.io/projected/856f7d39-9e42-4bbc-afaa-58ef30a34aac-kube-api-access-kssnn\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:32.780628 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.780593    2561 generic.go:356] "Generic (PLEG): container finished" podID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" containerID="67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354" exitCode=0
Jul 06 19:21:32.781036 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.780636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"
Jul 06 19:21:32.781036 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.780684    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" event={"ID":"856f7d39-9e42-4bbc-afaa-58ef30a34aac","Type":"ContainerDied","Data":"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"}
Jul 06 19:21:32.781036 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.780742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn" event={"ID":"856f7d39-9e42-4bbc-afaa-58ef30a34aac","Type":"ContainerDied","Data":"76be6f84df54150a435cb54c24d0cd3e0a32fe8267024ab3d8b728ade1a31fa6"}
Jul 06 19:21:32.781036 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.780766    2561 scope.go:117] "RemoveContainer" containerID="67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"
Jul 06 19:21:32.790335 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.790317    2561 scope.go:117] "RemoveContainer" containerID="67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"
Jul 06 19:21:32.790657 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:32.790635    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354\": container with ID starting with 67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354 not found: ID does not exist" containerID="67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"
Jul 06 19:21:32.790700 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.790668    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354"} err="failed to get container status \"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354\": rpc error: code = NotFound desc = could not find container \"67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354\": container with ID starting with 67f186043aad146fc6ca97e093af526e8736e18950aa90e95e47441712864354 not found: ID does not exist"
Jul 06 19:21:32.809244 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.809217    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:32.814409 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:32.814387    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-1bd5e6c9-6c5d6c459d-bvgdn"]
Jul 06 19:21:33.963970 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:33.963901    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" path="/var/lib/kubelet/pods/856f7d39-9e42-4bbc-afaa-58ef30a34aac/volumes"
Jul 06 19:21:39.835958 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.835925    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:39.836426 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.836265    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" containerName="maas-api"
Jul 06 19:21:39.836426 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.836277    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" containerName="maas-api"
Jul 06 19:21:39.836426 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.836337    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="856f7d39-9e42-4bbc-afaa-58ef30a34aac" containerName="maas-api"
Jul 06 19:21:39.840729 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.840708    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:39.844432 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.844412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-8e717861-serving-cert\""
Jul 06 19:21:39.870338 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.870306    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:39.921836 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.921805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:39.922068 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.921853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vh2\" (UniqueName: \"kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:39.922068 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:39.921878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.022856 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.022820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.022856 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.022873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vh2\" (UniqueName: \"kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.023136 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.022890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.023136 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:40.023018    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-8e717861-serving-cert: secret "maas-api-e2e-ait-8e717861-serving-cert" not found
Jul 06 19:21:40.023136 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:40.023073    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls podName:3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f nodeName:}" failed. No retries permitted until 2026-07-06 19:21:40.52305621 +0000 UTC m=+1603.109264163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls") pod "maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" (UID: "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f") : secret "maas-api-e2e-ait-8e717861-serving-cert" not found
Jul 06 19:21:40.023600 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.023580    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.040669 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.040633    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vh2\" (UniqueName: \"kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.527857 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.527819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.530938 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.530911    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") pod \"maas-api-e2e-ait-8e717861-68b77ff44-sc5v6\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") " pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.752147 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.752107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:40.911308 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:40.911261    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:40.918221 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:21:40.918191    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8586fc_c308_4ba3_8777_ef2ef6c6aa5f.slice/crio-76dfdea25184bd6555139f78e6e2109c9cfa86106d924ddb76eedf2a27c7c9ff WatchSource:0}: Error finding container 76dfdea25184bd6555139f78e6e2109c9cfa86106d924ddb76eedf2a27c7c9ff: Status 404 returned error can't find the container with id 76dfdea25184bd6555139f78e6e2109c9cfa86106d924ddb76eedf2a27c7c9ff
Jul 06 19:21:41.826225 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:41.826184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" event={"ID":"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f","Type":"ContainerStarted","Data":"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"}
Jul 06 19:21:41.826404 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:41.826238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" event={"ID":"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f","Type":"ContainerStarted","Data":"76dfdea25184bd6555139f78e6e2109c9cfa86106d924ddb76eedf2a27c7c9ff"}
Jul 06 19:21:41.826404 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:41.826331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:41.869545 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:41.869492    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" podStartSLOduration=2.465735666 podStartE2EDuration="2.86945554s" totalImagesPullingTime="403.719874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:41.868119272 +0000 UTC m=+1604.454327246" watchObservedRunningTime="2026-07-06 19:21:41.86945554 +0000 UTC m=+1604.455663555"
Jul 06 19:21:45.193820 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.193783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:45.194297 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.194033    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" podUID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" containerName="maas-api" containerID="cri-o://3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c" gracePeriod=30
Jul 06 19:21:45.454581 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.454517    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:45.575432 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.575395    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca\") pod \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") "
Jul 06 19:21:45.575641 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.575518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") pod \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") "
Jul 06 19:21:45.575641 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.575592    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8vh2\" (UniqueName: \"kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2\") pod \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\" (UID: \"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f\") "
Jul 06 19:21:45.575816 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.575790    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" (UID: "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:21:45.577899 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.577862    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" (UID: "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:21:45.577899 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.577874    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2" (OuterVolumeSpecName: "kube-api-access-g8vh2") pod "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" (UID: "3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f"). InnerVolumeSpecName "kube-api-access-g8vh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:21:45.677163 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.677120    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vh2\" (UniqueName: \"kubernetes.io/projected/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-kube-api-access-g8vh2\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:45.677163 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.677155    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:45.677163 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.677164    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:21:45.845618 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.845584    2561 generic.go:356] "Generic (PLEG): container finished" podID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" containerID="3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c" exitCode=0
Jul 06 19:21:45.845805 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.845637    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"
Jul 06 19:21:45.845805 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.845665    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" event={"ID":"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f","Type":"ContainerDied","Data":"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"}
Jul 06 19:21:45.845805 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.845707    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6" event={"ID":"3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f","Type":"ContainerDied","Data":"76dfdea25184bd6555139f78e6e2109c9cfa86106d924ddb76eedf2a27c7c9ff"}
Jul 06 19:21:45.845805 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.845731    2561 scope.go:117] "RemoveContainer" containerID="3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"
Jul 06 19:21:45.855112 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.855093    2561 scope.go:117] "RemoveContainer" containerID="3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"
Jul 06 19:21:45.855413 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:45.855390    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c\": container with ID starting with 3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c not found: ID does not exist" containerID="3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"
Jul 06 19:21:45.855511 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.855421    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c"} err="failed to get container status \"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c\": rpc error: code = NotFound desc = could not find container \"3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c\": container with ID starting with 3f6368a97556cbcefb1cfeb9e2b1c6984403960cb565d92cf9a28c06ca12e97c not found: ID does not exist"
Jul 06 19:21:45.872437 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.872403    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:45.882709 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.882684    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-8e717861-68b77ff44-sc5v6"]
Jul 06 19:21:45.962000 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:45.961960    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" path="/var/lib/kubelet/pods/3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f/volumes"
Jul 06 19:21:59.034698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.034660    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:21:59.035255 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.035233    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" containerName="maas-api"
Jul 06 19:21:59.035329 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.035258    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" containerName="maas-api"
Jul 06 19:21:59.035387 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.035338    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8586fc-c308-4ba3-8777-ef2ef6c6aa5f" containerName="maas-api"
Jul 06 19:21:59.040082 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.040060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.063525 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.063500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-1faecda1-serving-cert\""
Jul 06 19:21:59.085741 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.085709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.085927 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.085758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.085927 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.085803    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnss6\" (UniqueName: \"kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.086208 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.086183    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:21:59.186620 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.186588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.186813 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.186638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.186813 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.186671    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnss6\" (UniqueName: \"kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.186813 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:59.186758    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-1faecda1-serving-cert: secret "maas-api-e2e-derive-1faecda1-serving-cert" not found
Jul 06 19:21:59.186926 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:21:59.186836    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls podName:df722715-a437-47f5-b922-ab22a635bd89 nodeName:}" failed. No retries permitted until 2026-07-06 19:21:59.686814092 +0000 UTC m=+1622.273022045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls") pod "maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" (UID: "df722715-a437-47f5-b922-ab22a635bd89") : secret "maas-api-e2e-derive-1faecda1-serving-cert" not found
Jul 06 19:21:59.187345 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.187325    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.222820 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.222791    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnss6\" (UniqueName: \"kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.690592 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.690554    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.693293 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.693259    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") pod \"maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") " pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:21:59.950455 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:21:59.950354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:22:00.100113 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:00.100083    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:22:00.103435 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:22:00.103401    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf722715_a437_47f5_b922_ab22a635bd89.slice/crio-01ebf5beb21e267f5f8d009c9a966b580e918a6a605ee2ee478acd938b9e1b23 WatchSource:0}: Error finding container 01ebf5beb21e267f5f8d009c9a966b580e918a6a605ee2ee478acd938b9e1b23: Status 404 returned error can't find the container with id 01ebf5beb21e267f5f8d009c9a966b580e918a6a605ee2ee478acd938b9e1b23
Jul 06 19:22:00.916127 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:00.916093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" event={"ID":"df722715-a437-47f5-b922-ab22a635bd89","Type":"ContainerStarted","Data":"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"}
Jul 06 19:22:00.916322 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:00.916145    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" event={"ID":"df722715-a437-47f5-b922-ab22a635bd89","Type":"ContainerStarted","Data":"01ebf5beb21e267f5f8d009c9a966b580e918a6a605ee2ee478acd938b9e1b23"}
Jul 06 19:22:00.916322 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:00.916233    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:22:00.940833 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:00.940780    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" podStartSLOduration=1.4927298580000001 podStartE2EDuration="1.940761514s" totalImagesPullingTime="448.031656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:22:00.938713427 +0000 UTC m=+1623.524921403" watchObservedRunningTime="2026-07-06 19:22:00.940761514 +0000 UTC m=+1623.526969522"
Jul 06 19:22:03.936394 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:03.936360    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:22:03.936908 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:03.936695    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" podUID="df722715-a437-47f5-b922-ab22a635bd89" containerName="maas-api" containerID="cri-o://4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795" gracePeriod=30
Jul 06 19:22:04.187765 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.187697    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:22:04.226004 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.225973    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnss6\" (UniqueName: \"kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6\") pod \"df722715-a437-47f5-b922-ab22a635bd89\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") "
Jul 06 19:22:04.226164 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.226031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca\") pod \"df722715-a437-47f5-b922-ab22a635bd89\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") "
Jul 06 19:22:04.226164 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.226134    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") pod \"df722715-a437-47f5-b922-ab22a635bd89\" (UID: \"df722715-a437-47f5-b922-ab22a635bd89\") "
Jul 06 19:22:04.226402 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.226380    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df722715-a437-47f5-b922-ab22a635bd89" (UID: "df722715-a437-47f5-b922-ab22a635bd89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:22:04.228300 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.228265    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6" (OuterVolumeSpecName: "kube-api-access-qnss6") pod "df722715-a437-47f5-b922-ab22a635bd89" (UID: "df722715-a437-47f5-b922-ab22a635bd89"). InnerVolumeSpecName "kube-api-access-qnss6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:22:04.228438 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.228355    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df722715-a437-47f5-b922-ab22a635bd89" (UID: "df722715-a437-47f5-b922-ab22a635bd89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:22:04.327125 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.327086    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df722715-a437-47f5-b922-ab22a635bd89-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:22:04.327125 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.327123    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df722715-a437-47f5-b922-ab22a635bd89-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:22:04.327125 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.327132    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnss6\" (UniqueName: \"kubernetes.io/projected/df722715-a437-47f5-b922-ab22a635bd89-kube-api-access-qnss6\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:22:04.934279 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.934229    2561 generic.go:356] "Generic (PLEG): container finished" podID="df722715-a437-47f5-b922-ab22a635bd89" containerID="4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795" exitCode=0
Jul 06 19:22:04.934279 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.934271    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"
Jul 06 19:22:04.934586 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.934307    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" event={"ID":"df722715-a437-47f5-b922-ab22a635bd89","Type":"ContainerDied","Data":"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"}
Jul 06 19:22:04.934586 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.934363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7" event={"ID":"df722715-a437-47f5-b922-ab22a635bd89","Type":"ContainerDied","Data":"01ebf5beb21e267f5f8d009c9a966b580e918a6a605ee2ee478acd938b9e1b23"}
Jul 06 19:22:04.934586 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.934390    2561 scope.go:117] "RemoveContainer" containerID="4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"
Jul 06 19:22:04.944590 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.944435    2561 scope.go:117] "RemoveContainer" containerID="4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"
Jul 06 19:22:04.944858 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:22:04.944762    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795\": container with ID starting with 4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795 not found: ID does not exist" containerID="4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"
Jul 06 19:22:04.944858 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.944805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795"} err="failed to get container status \"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795\": rpc error: code = NotFound desc = could not find container \"4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795\": container with ID starting with 4960e8b03af306bb52f4da78fce4e1d65342e79613cec84cbb9890aee686a795 not found: ID does not exist"
Jul 06 19:22:04.959510 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.959483    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:22:04.968671 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:04.968644    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-1faecda1-7c49f488c4-dfrh7"]
Jul 06 19:22:05.962430 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:22:05.962391    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df722715-a437-47f5-b922-ab22a635bd89" path="/var/lib/kubelet/pods/df722715-a437-47f5-b922-ab22a635bd89/volumes"
Jul 06 19:23:00.500443 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:00.500394    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" podUID="a6aab718-837a-4515-80a6-7379f5b5b0f8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 19:23:58.073564 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.073507    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:23:58.074134 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.074113    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df722715-a437-47f5-b922-ab22a635bd89" containerName="maas-api"
Jul 06 19:23:58.074190 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.074140    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df722715-a437-47f5-b922-ab22a635bd89" containerName="maas-api"
Jul 06 19:23:58.074230 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.074223    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="df722715-a437-47f5-b922-ab22a635bd89" containerName="maas-api"
Jul 06 19:23:58.077677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.077654    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.081910 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.081879    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a5970fbd-serving-cert\""
Jul 06 19:23:58.114134 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.114094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.114326 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.114137    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsdx\" (UniqueName: \"kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.114326 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.114230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.149845 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.149806    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:23:58.214591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.214552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsdx\" (UniqueName: \"kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.214811 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.214623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.214811 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.214662    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.214811 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:23:58.214746    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-a5970fbd-serving-cert: secret "maas-api-e2e-mt-a5970fbd-serving-cert" not found
Jul 06 19:23:58.214811 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:23:58.214815    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls podName:871efd06-df1b-42e0-9e2d-42cbcf54f888 nodeName:}" failed. No retries permitted until 2026-07-06 19:23:58.714796348 +0000 UTC m=+1741.301004299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls") pod "maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" (UID: "871efd06-df1b-42e0-9e2d-42cbcf54f888") : secret "maas-api-e2e-mt-a5970fbd-serving-cert" not found
Jul 06 19:23:58.215476 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.215433    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.242171 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.242135    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsdx\" (UniqueName: \"kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.716843 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.716806    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.719618 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.719581    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") pod \"maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") " pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:58.991031 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:58.990935    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:23:59.142521 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:59.142497    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:23:59.145359 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:23:59.145330    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871efd06_df1b_42e0_9e2d_42cbcf54f888.slice/crio-c911aa46edee5a32c1c2093eebda537e8f83ab1c070134887e6def8f0601d8b2 WatchSource:0}: Error finding container c911aa46edee5a32c1c2093eebda537e8f83ab1c070134887e6def8f0601d8b2: Status 404 returned error can't find the container with id c911aa46edee5a32c1c2093eebda537e8f83ab1c070134887e6def8f0601d8b2
Jul 06 19:23:59.435023 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:23:59.434976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" event={"ID":"871efd06-df1b-42e0-9e2d-42cbcf54f888","Type":"ContainerStarted","Data":"c911aa46edee5a32c1c2093eebda537e8f83ab1c070134887e6def8f0601d8b2"}
Jul 06 19:24:00.441760 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:00.441721    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" event={"ID":"871efd06-df1b-42e0-9e2d-42cbcf54f888","Type":"ContainerStarted","Data":"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"}
Jul 06 19:24:00.442256 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:00.441893    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:24:00.485048 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:00.484999    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" podStartSLOduration=2.05601308 podStartE2EDuration="2.484981717s" totalImagesPullingTime="428.968637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:00.484309688 +0000 UTC m=+1743.070517663" watchObservedRunningTime="2026-07-06 19:24:00.484981717 +0000 UTC m=+1743.071189739"
Jul 06 19:24:06.453326 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:06.453291    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:24:11.076640 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.076602    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:24:11.077123 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.076891    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" podUID="871efd06-df1b-42e0-9e2d-42cbcf54f888" containerName="maas-api" containerID="cri-o://9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3" gracePeriod=30
Jul 06 19:24:11.334847 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.334781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:24:11.409606 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.409568    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljsdx\" (UniqueName: \"kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx\") pod \"871efd06-df1b-42e0-9e2d-42cbcf54f888\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") "
Jul 06 19:24:11.409606 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.409610    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca\") pod \"871efd06-df1b-42e0-9e2d-42cbcf54f888\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") "
Jul 06 19:24:11.409843 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.409664    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") pod \"871efd06-df1b-42e0-9e2d-42cbcf54f888\" (UID: \"871efd06-df1b-42e0-9e2d-42cbcf54f888\") "
Jul 06 19:24:11.410010 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.409984    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "871efd06-df1b-42e0-9e2d-42cbcf54f888" (UID: "871efd06-df1b-42e0-9e2d-42cbcf54f888"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:24:11.411932 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.411907    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "871efd06-df1b-42e0-9e2d-42cbcf54f888" (UID: "871efd06-df1b-42e0-9e2d-42cbcf54f888"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:24:11.412034 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.411906    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx" (OuterVolumeSpecName: "kube-api-access-ljsdx") pod "871efd06-df1b-42e0-9e2d-42cbcf54f888" (UID: "871efd06-df1b-42e0-9e2d-42cbcf54f888"). InnerVolumeSpecName "kube-api-access-ljsdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:24:11.491377 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.491343    2561 generic.go:356] "Generic (PLEG): container finished" podID="871efd06-df1b-42e0-9e2d-42cbcf54f888" containerID="9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3" exitCode=0
Jul 06 19:24:11.491591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.491393    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"
Jul 06 19:24:11.491591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.491405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" event={"ID":"871efd06-df1b-42e0-9e2d-42cbcf54f888","Type":"ContainerDied","Data":"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"}
Jul 06 19:24:11.491591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.491451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj" event={"ID":"871efd06-df1b-42e0-9e2d-42cbcf54f888","Type":"ContainerDied","Data":"c911aa46edee5a32c1c2093eebda537e8f83ab1c070134887e6def8f0601d8b2"}
Jul 06 19:24:11.491591 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.491501    2561 scope.go:117] "RemoveContainer" containerID="9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"
Jul 06 19:24:11.500772 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.500754    2561 scope.go:117] "RemoveContainer" containerID="9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"
Jul 06 19:24:11.501034 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:24:11.501011    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3\": container with ID starting with 9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3 not found: ID does not exist" containerID="9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"
Jul 06 19:24:11.501102 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.501048    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3"} err="failed to get container status \"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3\": rpc error: code = NotFound desc = could not find container \"9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3\": container with ID starting with 9f71360dad052f297e01130ee01daadee86b299bece8683792e823d78e95c0c3 not found: ID does not exist"
Jul 06 19:24:11.511075 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.511049    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/871efd06-df1b-42e0-9e2d-42cbcf54f888-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:24:11.511075 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.511074    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljsdx\" (UniqueName: \"kubernetes.io/projected/871efd06-df1b-42e0-9e2d-42cbcf54f888-kube-api-access-ljsdx\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:24:11.511249 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.511084    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/871efd06-df1b-42e0-9e2d-42cbcf54f888-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:24:11.516850 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.516824    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:24:11.522903 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.522879    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a5970fbd-7459f5579d-gchhj"]
Jul 06 19:24:11.961913 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:11.961879    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871efd06-df1b-42e0-9e2d-42cbcf54f888" path="/var/lib/kubelet/pods/871efd06-df1b-42e0-9e2d-42cbcf54f888/volumes"
Jul 06 19:24:42.212100 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.212060    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:24:42.212677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.212590    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871efd06-df1b-42e0-9e2d-42cbcf54f888" containerName="maas-api"
Jul 06 19:24:42.212677 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.212606    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="871efd06-df1b-42e0-9e2d-42cbcf54f888" containerName="maas-api"
Jul 06 19:24:42.212753 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.212711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="871efd06-df1b-42e0-9e2d-42cbcf54f888" containerName="maas-api"
Jul 06 19:24:42.215744 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.215722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.218910 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.218885    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-4f73e900-serving-cert\""
Jul 06 19:24:42.280906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.280858    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:24:42.389068 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.389031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.389311 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.389095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.389311 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.389134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdmhw\" (UniqueName: \"kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.489784 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.489674    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.489784 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.489748    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.489784 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.489780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdmhw\" (UniqueName: \"kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.490099 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:24:42.489891    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-4f73e900-serving-cert: secret "maas-api-e2e-mt-4f73e900-serving-cert" not found
Jul 06 19:24:42.490099 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:24:42.489967    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls podName:b4f44009-9823-4cd7-bc54-6ee35167c533 nodeName:}" failed. No retries permitted until 2026-07-06 19:24:42.989949662 +0000 UTC m=+1785.576157613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls") pod "maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" (UID: "b4f44009-9823-4cd7-bc54-6ee35167c533") : secret "maas-api-e2e-mt-4f73e900-serving-cert" not found
Jul 06 19:24:42.490412 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.490387    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.527406 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.527366    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdmhw\" (UniqueName: \"kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.995409 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.995355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:42.998173 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:42.998137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") pod \"maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") " pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:43.127147 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:43.127103    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:43.302730 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:43.302701    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:24:43.304503 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:24:43.304452    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f44009_9823_4cd7_bc54_6ee35167c533.slice/crio-084387587c721d2dd131af8c17cf6030d247746e5f7a5636a251e2a28f0e3be3 WatchSource:0}: Error finding container 084387587c721d2dd131af8c17cf6030d247746e5f7a5636a251e2a28f0e3be3: Status 404 returned error can't find the container with id 084387587c721d2dd131af8c17cf6030d247746e5f7a5636a251e2a28f0e3be3
Jul 06 19:24:43.650650 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:43.650615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" event={"ID":"b4f44009-9823-4cd7-bc54-6ee35167c533","Type":"ContainerStarted","Data":"084387587c721d2dd131af8c17cf6030d247746e5f7a5636a251e2a28f0e3be3"}
Jul 06 19:24:44.657571 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:44.657533    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" event={"ID":"b4f44009-9823-4cd7-bc54-6ee35167c533","Type":"ContainerStarted","Data":"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"}
Jul 06 19:24:44.658017 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:44.657611    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:24:44.683747 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:44.683691    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" podStartSLOduration=2.304313141 podStartE2EDuration="2.683671134s" totalImagesPullingTime="379.357993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:24:44.680869895 +0000 UTC m=+1787.267077870" watchObservedRunningTime="2026-07-06 19:24:44.683671134 +0000 UTC m=+1787.269879108"
Jul 06 19:24:50.667762 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:24:50.667680    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:25:15.020426 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.020387    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:25:15.020959 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.020660    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" podUID="b4f44009-9823-4cd7-bc54-6ee35167c533" containerName="maas-api" containerID="cri-o://6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2" gracePeriod=30
Jul 06 19:25:15.294859 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.294832    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:25:15.371961 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.371925    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") pod \"b4f44009-9823-4cd7-bc54-6ee35167c533\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") "
Jul 06 19:25:15.372144 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.371991    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca\") pod \"b4f44009-9823-4cd7-bc54-6ee35167c533\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") "
Jul 06 19:25:15.372144 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.372018    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdmhw\" (UniqueName: \"kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw\") pod \"b4f44009-9823-4cd7-bc54-6ee35167c533\" (UID: \"b4f44009-9823-4cd7-bc54-6ee35167c533\") "
Jul 06 19:25:15.372388 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.372361    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b4f44009-9823-4cd7-bc54-6ee35167c533" (UID: "b4f44009-9823-4cd7-bc54-6ee35167c533"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:25:15.374318 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.374286    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b4f44009-9823-4cd7-bc54-6ee35167c533" (UID: "b4f44009-9823-4cd7-bc54-6ee35167c533"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:25:15.374391 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.374341    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw" (OuterVolumeSpecName: "kube-api-access-sdmhw") pod "b4f44009-9823-4cd7-bc54-6ee35167c533" (UID: "b4f44009-9823-4cd7-bc54-6ee35167c533"). InnerVolumeSpecName "kube-api-access-sdmhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:25:15.473014 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.472973    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f44009-9823-4cd7-bc54-6ee35167c533-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:25:15.473014 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.473012    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdmhw\" (UniqueName: \"kubernetes.io/projected/b4f44009-9823-4cd7-bc54-6ee35167c533-kube-api-access-sdmhw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:25:15.473014 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.473022    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4f44009-9823-4cd7-bc54-6ee35167c533-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:25:15.796779 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.796744    2561 generic.go:356] "Generic (PLEG): container finished" podID="b4f44009-9823-4cd7-bc54-6ee35167c533" containerID="6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2" exitCode=0
Jul 06 19:25:15.796966 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.796796    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"
Jul 06 19:25:15.796966 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.796808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" event={"ID":"b4f44009-9823-4cd7-bc54-6ee35167c533","Type":"ContainerDied","Data":"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"}
Jul 06 19:25:15.796966 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.796854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x" event={"ID":"b4f44009-9823-4cd7-bc54-6ee35167c533","Type":"ContainerDied","Data":"084387587c721d2dd131af8c17cf6030d247746e5f7a5636a251e2a28f0e3be3"}
Jul 06 19:25:15.796966 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.796890    2561 scope.go:117] "RemoveContainer" containerID="6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"
Jul 06 19:25:15.806936 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.806920    2561 scope.go:117] "RemoveContainer" containerID="6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"
Jul 06 19:25:15.807227 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:25:15.807207    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2\": container with ID starting with 6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2 not found: ID does not exist" containerID="6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"
Jul 06 19:25:15.807304 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.807240    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2"} err="failed to get container status \"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2\": rpc error: code = NotFound desc = could not find container \"6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2\": container with ID starting with 6cd60e34303c4b478277d228bd676ecbe7fc756129df2223faaf64cc3ab1c8a2 not found: ID does not exist"
Jul 06 19:25:15.841177 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.841144    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:25:15.847302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.847277    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-4f73e900-644fdd9b46-7vr2x"]
Jul 06 19:25:15.962063 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:25:15.962019    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f44009-9823-4cd7-bc54-6ee35167c533" path="/var/lib/kubelet/pods/b4f44009-9823-4cd7-bc54-6ee35167c533/volumes"
Jul 06 19:27:44.209007 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.208973    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:27:44.209448 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.209300    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f44009-9823-4cd7-bc54-6ee35167c533" containerName="maas-api"
Jul 06 19:27:44.209448 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.209313    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f44009-9823-4cd7-bc54-6ee35167c533" containerName="maas-api"
Jul 06 19:27:44.209448 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.209377    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f44009-9823-4cd7-bc54-6ee35167c533" containerName="maas-api"
Jul 06 19:27:44.212583 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.212552    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.216882 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.216854    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-88c5f6-serving-cert\""
Jul 06 19:27:44.235431 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.235399    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:27:44.287747 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.287692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.287942 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.287757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bk7c\" (UniqueName: \"kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.287942 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.287801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.389114 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.389070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.389114 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.389113    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bk7c\" (UniqueName: \"kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.389388 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.389146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.389388 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:27:44.389242    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-88c5f6-serving-cert: secret "maas-api-e2e-inf-a-88c5f6-serving-cert" not found
Jul 06 19:27:44.389388 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:27:44.389312    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls podName:c9d72e32-3ac0-43f9-a40b-1a952e8d46f2 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:44.88929203 +0000 UTC m=+1967.475499981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls") pod "maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" (UID: "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2") : secret "maas-api-e2e-inf-a-88c5f6-serving-cert" not found
Jul 06 19:27:44.389769 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.389747    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.426113 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.426076    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bk7c\" (UniqueName: \"kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.894427 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:44.894385    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:44.894628 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:27:44.894549    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-88c5f6-serving-cert: secret "maas-api-e2e-inf-a-88c5f6-serving-cert" not found
Jul 06 19:27:44.894628 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:27:44.894625    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls podName:c9d72e32-3ac0-43f9-a40b-1a952e8d46f2 nodeName:}" failed. No retries permitted until 2026-07-06 19:27:45.894606768 +0000 UTC m=+1968.480814725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls") pod "maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" (UID: "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2") : secret "maas-api-e2e-inf-a-88c5f6-serving-cert" not found
Jul 06 19:27:45.904285 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:45.904252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:45.906868 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:45.906834    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") " pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:46.023641 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:46.023602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:46.189496 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:46.189385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:27:46.192577 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:27:46.192543    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d72e32_3ac0_43f9_a40b_1a952e8d46f2.slice/crio-e0791f095d84e2220449d328d3626d3062e9f8c36417dac82f89432623abd2e5 WatchSource:0}: Error finding container e0791f095d84e2220449d328d3626d3062e9f8c36417dac82f89432623abd2e5: Status 404 returned error can't find the container with id e0791f095d84e2220449d328d3626d3062e9f8c36417dac82f89432623abd2e5
Jul 06 19:27:46.194127 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:46.194103    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 19:27:46.470068 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:46.469976    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" event={"ID":"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2","Type":"ContainerStarted","Data":"e0791f095d84e2220449d328d3626d3062e9f8c36417dac82f89432623abd2e5"}
Jul 06 19:27:47.476137 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:47.476102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" event={"ID":"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2","Type":"ContainerStarted","Data":"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"}
Jul 06 19:27:47.476584 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:47.476223    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:53.486798 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:53.486716    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:27:58.675863 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.675806    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" podStartSLOduration=14.246222867 podStartE2EDuration="14.675781773s" totalImagesPullingTime="429.558906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:27:47.520846496 +0000 UTC m=+1970.107054469" watchObservedRunningTime="2026-07-06 19:27:58.675781773 +0000 UTC m=+1981.261989752"
Jul 06 19:27:58.677676 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.677635    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:27:58.682394 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.682359    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.687072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.687046    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-cc274f-serving-cert\""
Jul 06 19:27:58.823942 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.823904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.823942 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.823946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzs8\" (UniqueName: \"kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.824171 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.823972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.924705 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.924663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.924705 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.924705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzs8\" (UniqueName: \"kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.924916 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.924734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.925339 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.925320    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:58.927394 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:58.927323    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:27:59.223810 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:59.223723    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:27:59.992871 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:27:59.992834    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzs8\" (UniqueName: \"kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8\") pod \"maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") " pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:28:00.195658 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:00.195620    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:28:05.027783 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:05.027739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:28:05.038568 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:28:05.038531    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93372cd_0796_4e9f_8a43_6267d1bcbf25.slice/crio-601b94a21cff26d9d1a45b7e8824c15d589b26a69f6df6716919554f17071b6b WatchSource:0}: Error finding container 601b94a21cff26d9d1a45b7e8824c15d589b26a69f6df6716919554f17071b6b: Status 404 returned error can't find the container with id 601b94a21cff26d9d1a45b7e8824c15d589b26a69f6df6716919554f17071b6b
Jul 06 19:28:05.557746 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:05.557711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" event={"ID":"e93372cd-0796-4e9f-8a43-6267d1bcbf25","Type":"ContainerStarted","Data":"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"}
Jul 06 19:28:05.557950 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:05.557765    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" event={"ID":"e93372cd-0796-4e9f-8a43-6267d1bcbf25","Type":"ContainerStarted","Data":"601b94a21cff26d9d1a45b7e8824c15d589b26a69f6df6716919554f17071b6b"}
Jul 06 19:28:05.557950 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:05.557802    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:28:08.689473 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:08.689410    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" podStartSLOduration=16.371154417 podStartE2EDuration="16.689392823s" totalImagesPullingTime="318.238406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:28:08.662326364 +0000 UTC m=+1991.248534338" watchObservedRunningTime="2026-07-06 19:28:08.689392823 +0000 UTC m=+1991.275600796"
Jul 06 19:28:10.499563 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:10.499521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" podUID="a6aab718-837a-4515-80a6-7379f5b5b0f8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 19:28:11.568993 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:28:11.568966    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:29:02.535398 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.535359    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:29:02.535970 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.535637    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" podUID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" containerName="maas-api" containerID="cri-o://ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4" gracePeriod=30
Jul 06 19:29:02.787977 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.787906    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:29:02.793453 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.793433    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bk7c\" (UniqueName: \"kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c\") pod \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") "
Jul 06 19:29:02.793598 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.793493    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") pod \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") "
Jul 06 19:29:02.793598 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.793549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca\") pod \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\" (UID: \"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2\") "
Jul 06 19:29:02.793978 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.793952    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" (UID: "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:29:02.795739 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.795711    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c" (OuterVolumeSpecName: "kube-api-access-9bk7c") pod "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" (UID: "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2"). InnerVolumeSpecName "kube-api-access-9bk7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:29:02.795819 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.795737    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" (UID: "c9d72e32-3ac0-43f9-a40b-1a952e8d46f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:29:02.828154 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.828117    2561 generic.go:356] "Generic (PLEG): container finished" podID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" containerID="ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4" exitCode=0
Jul 06 19:29:02.828306 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.828197    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"
Jul 06 19:29:02.828306 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.828231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" event={"ID":"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2","Type":"ContainerDied","Data":"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"}
Jul 06 19:29:02.828306 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.828280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z" event={"ID":"c9d72e32-3ac0-43f9-a40b-1a952e8d46f2","Type":"ContainerDied","Data":"e0791f095d84e2220449d328d3626d3062e9f8c36417dac82f89432623abd2e5"}
Jul 06 19:29:02.828306 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.828301    2561 scope.go:117] "RemoveContainer" containerID="ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"
Jul 06 19:29:02.843378 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.843340    2561 scope.go:117] "RemoveContainer" containerID="ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"
Jul 06 19:29:02.844064 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:29:02.844028    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4\": container with ID starting with ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4 not found: ID does not exist" containerID="ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"
Jul 06 19:29:02.844188 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.844076    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4"} err="failed to get container status \"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4\": rpc error: code = NotFound desc = could not find container \"ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4\": container with ID starting with ada809f3ade1c43f3e328ad653e2506c99d8cd888bb2edd8f479ae03f4d768d4 not found: ID does not exist"
Jul 06 19:29:02.858249 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.858221    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:29:02.866198 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.866171    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-88c5f6-865f8d55cb-7zm6z"]
Jul 06 19:29:02.894698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.894658    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:02.894698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.894691    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bk7c\" (UniqueName: \"kubernetes.io/projected/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-kube-api-access-9bk7c\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:02.894698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:02.894702    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:03.963671 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:03.963640    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" path="/var/lib/kubelet/pods/c9d72e32-3ac0-43f9-a40b-1a952e8d46f2/volumes"
Jul 06 19:29:46.133303 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.133266    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:29:46.135689 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.133560    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" podUID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" containerName="maas-api" containerID="cri-o://57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965" gracePeriod=30
Jul 06 19:29:46.377869 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.377844    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:29:46.479374 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.479291    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca\") pod \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") "
Jul 06 19:29:46.479374 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.479332    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxzs8\" (UniqueName: \"kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8\") pod \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") "
Jul 06 19:29:46.479593 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.479416    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls\") pod \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\" (UID: \"e93372cd-0796-4e9f-8a43-6267d1bcbf25\") "
Jul 06 19:29:46.479784 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.479758    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e93372cd-0796-4e9f-8a43-6267d1bcbf25" (UID: "e93372cd-0796-4e9f-8a43-6267d1bcbf25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:29:46.481817 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.481792    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e93372cd-0796-4e9f-8a43-6267d1bcbf25" (UID: "e93372cd-0796-4e9f-8a43-6267d1bcbf25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:29:46.481817 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.481803    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8" (OuterVolumeSpecName: "kube-api-access-qxzs8") pod "e93372cd-0796-4e9f-8a43-6267d1bcbf25" (UID: "e93372cd-0796-4e9f-8a43-6267d1bcbf25"). InnerVolumeSpecName "kube-api-access-qxzs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:29:46.580294 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.580253    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e93372cd-0796-4e9f-8a43-6267d1bcbf25-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:46.580294 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.580287    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93372cd-0796-4e9f-8a43-6267d1bcbf25-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:46.580294 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:46.580298    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzs8\" (UniqueName: \"kubernetes.io/projected/e93372cd-0796-4e9f-8a43-6267d1bcbf25-kube-api-access-qxzs8\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:29:47.027073 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.027036    2561 generic.go:356] "Generic (PLEG): container finished" podID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" containerID="57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965" exitCode=0
Jul 06 19:29:47.027264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.027084    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"
Jul 06 19:29:47.027264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.027122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" event={"ID":"e93372cd-0796-4e9f-8a43-6267d1bcbf25","Type":"ContainerDied","Data":"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"}
Jul 06 19:29:47.027264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.027179    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr" event={"ID":"e93372cd-0796-4e9f-8a43-6267d1bcbf25","Type":"ContainerDied","Data":"601b94a21cff26d9d1a45b7e8824c15d589b26a69f6df6716919554f17071b6b"}
Jul 06 19:29:47.027264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.027218    2561 scope.go:117] "RemoveContainer" containerID="57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"
Jul 06 19:29:47.040394 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.040368    2561 scope.go:117] "RemoveContainer" containerID="57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"
Jul 06 19:29:47.040817 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:29:47.040789    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965\": container with ID starting with 57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965 not found: ID does not exist" containerID="57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"
Jul 06 19:29:47.040920 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.040825    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965"} err="failed to get container status \"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965\": rpc error: code = NotFound desc = could not find container \"57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965\": container with ID starting with 57a391893d7aeb653e275da8b1dd85b1db185cd53a967396da3755e125ede965 not found: ID does not exist"
Jul 06 19:29:47.089753 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.089720    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:29:47.098997 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.098971    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-cc274f-6c66d49d48-vrmtr"]
Jul 06 19:29:47.966837 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:29:47.966802    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" path="/var/lib/kubelet/pods/e93372cd-0796-4e9f-8a43-6267d1bcbf25/volumes"
Jul 06 19:30:11.522066 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522030    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522396    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" containerName="maas-api"
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522408    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" containerName="maas-api"
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522422    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" containerName="maas-api"
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522427    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" containerName="maas-api"
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522501    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e93372cd-0796-4e9f-8a43-6267d1bcbf25" containerName="maas-api"
Jul 06 19:30:11.522714 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.522513    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9d72e32-3ac0-43f9-a40b-1a952e8d46f2" containerName="maas-api"
Jul 06 19:30:11.525780 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.525756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.529149 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.529125    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-ebed37-serving-cert\""
Jul 06 19:30:11.578344 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.578307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:11.595474 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.595435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.595655 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.595504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqr5c\" (UniqueName: \"kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.595655 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.595628    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.696795 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.696751    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqr5c\" (UniqueName: \"kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.696996 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.696804    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.696996 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.696904    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.697089 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:30:11.697005    2561 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-ebed37-serving-cert: secret "maas-api-e2e-api-a-ebed37-serving-cert" not found
Jul 06 19:30:11.697089 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:30:11.697066    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls podName:788e5092-5916-4c40-8c87-4c9bdd2db04e nodeName:}" failed. No retries permitted until 2026-07-06 19:30:12.197049467 +0000 UTC m=+2114.783257419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls") pod "maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" (UID: "788e5092-5916-4c40-8c87-4c9bdd2db04e") : secret "maas-api-e2e-api-a-ebed37-serving-cert" not found
Jul 06 19:30:11.697553 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.697534    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:11.729094 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:11.729055    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqr5c\" (UniqueName: \"kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:12.201644 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:12.201605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:12.204390 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:12.204356    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") pod \"maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") " pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:12.437334 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:12.437291    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:12.623905 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:12.623876    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:12.624744 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:30:12.624717    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788e5092_5916_4c40_8c87_4c9bdd2db04e.slice/crio-a42c58b2762501a5cbe9ed8c5c5a0f87066b392ca1bc6716747968c46fb96175 WatchSource:0}: Error finding container a42c58b2762501a5cbe9ed8c5c5a0f87066b392ca1bc6716747968c46fb96175: Status 404 returned error can't find the container with id a42c58b2762501a5cbe9ed8c5c5a0f87066b392ca1bc6716747968c46fb96175
Jul 06 19:30:13.153576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:13.153546    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" event={"ID":"788e5092-5916-4c40-8c87-4c9bdd2db04e","Type":"ContainerStarted","Data":"a42c58b2762501a5cbe9ed8c5c5a0f87066b392ca1bc6716747968c46fb96175"}
Jul 06 19:30:14.160450 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:14.160405    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" event={"ID":"788e5092-5916-4c40-8c87-4c9bdd2db04e","Type":"ContainerStarted","Data":"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"}
Jul 06 19:30:14.160930 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:14.160644    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:20.170918 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.170882    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:20.239828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.239773    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" podStartSLOduration=8.712507229 podStartE2EDuration="9.23975293s" totalImagesPullingTime="527.245701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:14.215779308 +0000 UTC m=+2116.801987282" watchObservedRunningTime="2026-07-06 19:30:20.23975293 +0000 UTC m=+2122.825960956"
Jul 06 19:30:20.407230 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.407187    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:20.410626 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.410604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.432319 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.432250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-a30d6b-serving-cert\""
Jul 06 19:30:20.466841 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.466807    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:20.473315 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.473286    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.473570 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.473337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ddsw\" (UniqueName: \"kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.473570 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.473419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.574724 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.574684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.574904 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.574771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.574904 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.574808    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ddsw\" (UniqueName: \"kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.575532 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.575503    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.577375 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.577352    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.595499 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.595453    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ddsw\" (UniqueName: \"kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw\") pod \"maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") " pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.720921 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.720827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:20.906110 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:20.906085    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:20.906834 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:30:20.906806    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85603101_03af_4699_a2fd_3328dd9d585d.slice/crio-5362b3529d2188115a5317ab35c2f89f99d3f290bb6b5c194c7992c79a1d6df5 WatchSource:0}: Error finding container 5362b3529d2188115a5317ab35c2f89f99d3f290bb6b5c194c7992c79a1d6df5: Status 404 returned error can't find the container with id 5362b3529d2188115a5317ab35c2f89f99d3f290bb6b5c194c7992c79a1d6df5
Jul 06 19:30:21.192646 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:21.192612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" event={"ID":"85603101-03af-4699-a2fd-3328dd9d585d","Type":"ContainerStarted","Data":"5362b3529d2188115a5317ab35c2f89f99d3f290bb6b5c194c7992c79a1d6df5"}
Jul 06 19:30:22.198830 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:22.198788    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" event={"ID":"85603101-03af-4699-a2fd-3328dd9d585d","Type":"ContainerStarted","Data":"235708252a7374d574df2578212de028fe2204f0d115be8617fc646c522ac5d9"}
Jul 06 19:30:22.199242 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:22.198854    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:28.209142 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:28.209114    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:28.235310 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:28.235247    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" podStartSLOduration=7.811860394 podStartE2EDuration="8.235223425s" totalImagesPullingTime="423.363031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:22.247532621 +0000 UTC m=+2124.833740595" watchObservedRunningTime="2026-07-06 19:30:28.235223425 +0000 UTC m=+2130.821431392"
Jul 06 19:30:33.699673 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:33.699639    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:33.700073 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:33.699896    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" podUID="788e5092-5916-4c40-8c87-4c9bdd2db04e" containerName="maas-api" containerID="cri-o://4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7" gracePeriod=30
Jul 06 19:30:33.940580 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:33.940550    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:34.100452 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.100418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqr5c\" (UniqueName: \"kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c\") pod \"788e5092-5916-4c40-8c87-4c9bdd2db04e\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") "
Jul 06 19:30:34.100452 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.100453    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca\") pod \"788e5092-5916-4c40-8c87-4c9bdd2db04e\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") "
Jul 06 19:30:34.100698 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.100516    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") pod \"788e5092-5916-4c40-8c87-4c9bdd2db04e\" (UID: \"788e5092-5916-4c40-8c87-4c9bdd2db04e\") "
Jul 06 19:30:34.100922 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.100892    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "788e5092-5916-4c40-8c87-4c9bdd2db04e" (UID: "788e5092-5916-4c40-8c87-4c9bdd2db04e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:30:34.102667 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.102641    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "788e5092-5916-4c40-8c87-4c9bdd2db04e" (UID: "788e5092-5916-4c40-8c87-4c9bdd2db04e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:30:34.102755 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.102680    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c" (OuterVolumeSpecName: "kube-api-access-qqr5c") pod "788e5092-5916-4c40-8c87-4c9bdd2db04e" (UID: "788e5092-5916-4c40-8c87-4c9bdd2db04e"). InnerVolumeSpecName "kube-api-access-qqr5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:30:34.201558 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.201516    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqr5c\" (UniqueName: \"kubernetes.io/projected/788e5092-5916-4c40-8c87-4c9bdd2db04e-kube-api-access-qqr5c\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:34.201558 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.201548    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/788e5092-5916-4c40-8c87-4c9bdd2db04e-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:34.201558 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.201557    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/788e5092-5916-4c40-8c87-4c9bdd2db04e-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:34.252395 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.252361    2561 generic.go:356] "Generic (PLEG): container finished" podID="788e5092-5916-4c40-8c87-4c9bdd2db04e" containerID="4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7" exitCode=0
Jul 06 19:30:34.252577 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.252411    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"
Jul 06 19:30:34.252577 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.252451    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" event={"ID":"788e5092-5916-4c40-8c87-4c9bdd2db04e","Type":"ContainerDied","Data":"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"}
Jul 06 19:30:34.252577 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.252524    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs" event={"ID":"788e5092-5916-4c40-8c87-4c9bdd2db04e","Type":"ContainerDied","Data":"a42c58b2762501a5cbe9ed8c5c5a0f87066b392ca1bc6716747968c46fb96175"}
Jul 06 19:30:34.252577 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.252552    2561 scope.go:117] "RemoveContainer" containerID="4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"
Jul 06 19:30:34.261848 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.261830    2561 scope.go:117] "RemoveContainer" containerID="4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"
Jul 06 19:30:34.262151 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:30:34.262130    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7\": container with ID starting with 4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7 not found: ID does not exist" containerID="4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"
Jul 06 19:30:34.262208 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.262161    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7"} err="failed to get container status \"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7\": rpc error: code = NotFound desc = could not find container \"4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7\": container with ID starting with 4dd840baf07fbef08418d51c7dafc6001f9d79e72e22967be68ebb81dd0494b7 not found: ID does not exist"
Jul 06 19:30:34.275038 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.275007    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:34.281476 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:34.281438    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-ebed37-68fd5f65bc-mpzcs"]
Jul 06 19:30:35.962891 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:35.962854    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="788e5092-5916-4c40-8c87-4c9bdd2db04e" path="/var/lib/kubelet/pods/788e5092-5916-4c40-8c87-4c9bdd2db04e/volumes"
Jul 06 19:30:41.150775 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.150733    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:41.151268 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.150995    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" podUID="85603101-03af-4699-a2fd-3328dd9d585d" containerName="maas-api" containerID="cri-o://235708252a7374d574df2578212de028fe2204f0d115be8617fc646c522ac5d9" gracePeriod=30
Jul 06 19:30:41.285326 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.285290    2561 generic.go:356] "Generic (PLEG): container finished" podID="85603101-03af-4699-a2fd-3328dd9d585d" containerID="235708252a7374d574df2578212de028fe2204f0d115be8617fc646c522ac5d9" exitCode=0
Jul 06 19:30:41.285514 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.285365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" event={"ID":"85603101-03af-4699-a2fd-3328dd9d585d","Type":"ContainerDied","Data":"235708252a7374d574df2578212de028fe2204f0d115be8617fc646c522ac5d9"}
Jul 06 19:30:41.407252 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.407179    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:41.567962 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.567920    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls\") pod \"85603101-03af-4699-a2fd-3328dd9d585d\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") "
Jul 06 19:30:41.568202 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.568003    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ddsw\" (UniqueName: \"kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw\") pod \"85603101-03af-4699-a2fd-3328dd9d585d\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") "
Jul 06 19:30:41.568202 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.568104    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca\") pod \"85603101-03af-4699-a2fd-3328dd9d585d\" (UID: \"85603101-03af-4699-a2fd-3328dd9d585d\") "
Jul 06 19:30:41.568393 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.568372    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85603101-03af-4699-a2fd-3328dd9d585d" (UID: "85603101-03af-4699-a2fd-3328dd9d585d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:30:41.570287 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.570259    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw" (OuterVolumeSpecName: "kube-api-access-2ddsw") pod "85603101-03af-4699-a2fd-3328dd9d585d" (UID: "85603101-03af-4699-a2fd-3328dd9d585d"). InnerVolumeSpecName "kube-api-access-2ddsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:30:41.570390 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.570299    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85603101-03af-4699-a2fd-3328dd9d585d" (UID: "85603101-03af-4699-a2fd-3328dd9d585d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:30:41.669072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.668975    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ddsw\" (UniqueName: \"kubernetes.io/projected/85603101-03af-4699-a2fd-3328dd9d585d-kube-api-access-2ddsw\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:41.669072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.669012    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85603101-03af-4699-a2fd-3328dd9d585d-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:41.669072 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:41.669022    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85603101-03af-4699-a2fd-3328dd9d585d-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:30:42.290881 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:42.290800    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"
Jul 06 19:30:42.290881 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:42.290805    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh" event={"ID":"85603101-03af-4699-a2fd-3328dd9d585d","Type":"ContainerDied","Data":"5362b3529d2188115a5317ab35c2f89f99d3f290bb6b5c194c7992c79a1d6df5"}
Jul 06 19:30:42.290881 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:42.290870    2561 scope.go:117] "RemoveContainer" containerID="235708252a7374d574df2578212de028fe2204f0d115be8617fc646c522ac5d9"
Jul 06 19:30:42.309837 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:42.309803    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:42.314327 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:42.314299    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a30d6b-6649f8d654-bs9rh"]
Jul 06 19:30:43.963525 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:43.963492    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85603101-03af-4699-a2fd-3328dd9d585d" path="/var/lib/kubelet/pods/85603101-03af-4699-a2fd-3328dd9d585d/volumes"
Jul 06 19:30:52.507316 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507218    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507569    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="788e5092-5916-4c40-8c87-4c9bdd2db04e" containerName="maas-api"
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507581    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="788e5092-5916-4c40-8c87-4c9bdd2db04e" containerName="maas-api"
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507591    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85603101-03af-4699-a2fd-3328dd9d585d" containerName="maas-api"
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507597    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="85603101-03af-4699-a2fd-3328dd9d585d" containerName="maas-api"
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507660    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="85603101-03af-4699-a2fd-3328dd9d585d" containerName="maas-api"
Jul 06 19:30:52.507801 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.507667    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="788e5092-5916-4c40-8c87-4c9bdd2db04e" containerName="maas-api"
Jul 06 19:30:52.512084 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.512061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.514874 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.514847    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-3550d9-serving-cert\""
Jul 06 19:30:52.530594 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.530567    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:30:52.558304 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.558253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.558553 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.558526    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.558639 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.558622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5tx\" (UniqueName: \"kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.659119 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.659086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.659315 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.659141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5tx\" (UniqueName: \"kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.659315 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.659187    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.659908 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.659883    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.661748 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.661724    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.682130 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.682101    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5tx\" (UniqueName: \"kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx\") pod \"maas-api-e2e-shared-a-3550d9-9568c77dd-545l2\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") " pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:52.822526 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:52.822496    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:53.007605 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:30:53.007573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3226313_f8df_4847_8fc4_2942bfd92484.slice/crio-e94ac6b972362a9fc3024164069ee36ac2caefcb9838f251a2b5eb90130d6e20 WatchSource:0}: Error finding container e94ac6b972362a9fc3024164069ee36ac2caefcb9838f251a2b5eb90130d6e20: Status 404 returned error can't find the container with id e94ac6b972362a9fc3024164069ee36ac2caefcb9838f251a2b5eb90130d6e20
Jul 06 19:30:53.014035 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:53.014004    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:30:53.342906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:53.342871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" event={"ID":"f3226313-f8df-4847-8fc4-2942bfd92484","Type":"ContainerStarted","Data":"e94ac6b972362a9fc3024164069ee36ac2caefcb9838f251a2b5eb90130d6e20"}
Jul 06 19:30:54.348657 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:54.348621    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" event={"ID":"f3226313-f8df-4847-8fc4-2942bfd92484","Type":"ContainerStarted","Data":"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"}
Jul 06 19:30:54.349053 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:54.348735    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:30:54.377100 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:30:54.377045    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" podStartSLOduration=2.025073335 podStartE2EDuration="2.377028037s" totalImagesPullingTime="351.954702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:30:54.37466688 +0000 UTC m=+2156.960874854" watchObservedRunningTime="2026-07-06 19:30:54.377028037 +0000 UTC m=+2156.963236011"
Jul 06 19:31:00.359036 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.359000    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:31:00.590588 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.590553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:31:00.595543 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.595514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.599735 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.599708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-06f95c-serving-cert\""
Jul 06 19:31:00.626259 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.626156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.626259 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.626198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.626259 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.626216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sc6z\" (UniqueName: \"kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.673611 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.673577    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:31:00.727078 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.727035    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.727295 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.727086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.727295 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.727116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sc6z\" (UniqueName: \"kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.727869 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.727844    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.730099 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.730074    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.742397 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.742370    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sc6z\" (UniqueName: \"kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z\") pod \"maas-api-e2e-shared-b-06f95c-8685698754-97cqr\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") " pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:00.909956 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:00.909853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:01.099553 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:01.099368    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:31:01.101572 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:31:01.101542    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cc6167_1353_49e2_af47_bab82ed6c112.slice/crio-ec680a282d1eba97e26085db36bd05b82c644b40dbab2a2c45d9019cb558a879 WatchSource:0}: Error finding container ec680a282d1eba97e26085db36bd05b82c644b40dbab2a2c45d9019cb558a879: Status 404 returned error can't find the container with id ec680a282d1eba97e26085db36bd05b82c644b40dbab2a2c45d9019cb558a879
Jul 06 19:31:01.380239 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:01.380197    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" event={"ID":"c8cc6167-1353-49e2-af47-bab82ed6c112","Type":"ContainerStarted","Data":"ec680a282d1eba97e26085db36bd05b82c644b40dbab2a2c45d9019cb558a879"}
Jul 06 19:31:02.386744 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:02.386705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" event={"ID":"c8cc6167-1353-49e2-af47-bab82ed6c112","Type":"ContainerStarted","Data":"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"}
Jul 06 19:31:02.387189 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:02.386913    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:08.396910 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:08.396877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:31:08.442891 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:08.442835    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" podStartSLOduration=7.998394234 podStartE2EDuration="8.442815082s" totalImagesPullingTime="444.420848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:02.437438377 +0000 UTC m=+2165.023646384" watchObservedRunningTime="2026-07-06 19:31:08.442815082 +0000 UTC m=+2171.029023059"
Jul 06 19:31:38.111608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.111567    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:31:38.115050 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.115025    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.118323 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.118298    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"kube-root-ca.crt\""
Jul 06 19:31:38.118809 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.118790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"default-dockercfg-z7gx5\""
Jul 06 19:31:38.119260 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.119246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"auth-test-model-06f95c-kserve-self-signed-certs\""
Jul 06 19:31:38.124345 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.124317    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"openshift-service-ca.crt\""
Jul 06 19:31:38.125280 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.125255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:31:38.262061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.262251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.262251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.262251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.262251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262219    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.262251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.262240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfwt\" (UniqueName: \"kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.363864 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.363864 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364108 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364108 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363912    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364108 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364108 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.363957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfwt\" (UniqueName: \"kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.364298    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.364360    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.364490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.364380    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.366425 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.366402    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.366607 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.366590    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.373164 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.373142    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfwt\" (UniqueName: \"kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt\") pod \"auth-test-model-06f95c-kserve-5697c65b6c-mmqbv\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") " pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.426270 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.426230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:38.580908 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:38.580875    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:31:38.584143 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:31:38.584114    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefd811cf_595b_4a23_b70a_112e961a81d9.slice/crio-f70bcb0be2cecbb2c7fcb2085a4d785bd7dc7d149a18435dc2f8b0e6d033e202 WatchSource:0}: Error finding container f70bcb0be2cecbb2c7fcb2085a4d785bd7dc7d149a18435dc2f8b0e6d033e202: Status 404 returned error can't find the container with id f70bcb0be2cecbb2c7fcb2085a4d785bd7dc7d149a18435dc2f8b0e6d033e202
Jul 06 19:31:39.552748 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:39.552711    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" event={"ID":"efd811cf-595b-4a23-b70a-112e961a81d9","Type":"ContainerStarted","Data":"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"}
Jul 06 19:31:39.552748 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:39.552755    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" event={"ID":"efd811cf-595b-4a23-b70a-112e961a81d9","Type":"ContainerStarted","Data":"f70bcb0be2cecbb2c7fcb2085a4d785bd7dc7d149a18435dc2f8b0e6d033e202"}
Jul 06 19:31:39.569554 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:39.569499    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" podStartSLOduration=1.569427242 podStartE2EDuration="1.569427242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:31:39.568769138 +0000 UTC m=+2202.154977129" watchObservedRunningTime="2026-07-06 19:31:39.569427242 +0000 UTC m=+2202.155635215"
Jul 06 19:31:48.426662 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:48.426626    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:48.427167 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:48.426702    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:48.444699 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:48.444667    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:31:48.604993 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:31:48.604959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:34:01.923333 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:01.923249    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:34:16.945199 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:16.945135    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" podUID="efd811cf-595b-4a23-b70a-112e961a81d9" containerName="main" containerID="cri-o://cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253" gracePeriod=45
Jul 06 19:34:17.193220 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.193194    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:34:17.270157 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270057    2561 generic.go:356] "Generic (PLEG): container finished" podID="efd811cf-595b-4a23-b70a-112e961a81d9" containerID="cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253" exitCode=0
Jul 06 19:34:17.270157 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270101    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"
Jul 06 19:34:17.270157 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270123    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" event={"ID":"efd811cf-595b-4a23-b70a-112e961a81d9","Type":"ContainerDied","Data":"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"}
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270162    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv" event={"ID":"efd811cf-595b-4a23-b70a-112e961a81d9","Type":"ContainerDied","Data":"f70bcb0be2cecbb2c7fcb2085a4d785bd7dc7d149a18435dc2f8b0e6d033e202"}
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270184    2561 scope.go:117] "RemoveContainer" containerID="cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270281    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270331    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270349    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwfwt\" (UniqueName: \"kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270412    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270430    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs\") pod \"efd811cf-595b-4a23-b70a-112e961a81d9\" (UID: \"efd811cf-595b-4a23-b70a-112e961a81d9\") "
Jul 06 19:34:17.270828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270622    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home" (OuterVolumeSpecName: "home") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:34:17.270828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270784    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:34:17.271002 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.270877    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:34:17.273044 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.273013    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm" (OuterVolumeSpecName: "dshm") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:34:17.273044 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.273021    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:34:17.273044 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.273022    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt" (OuterVolumeSpecName: "kube-api-access-vwfwt") pod "efd811cf-595b-4a23-b70a-112e961a81d9" (UID: "efd811cf-595b-4a23-b70a-112e961a81d9"). InnerVolumeSpecName "kube-api-access-vwfwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:34:17.290643 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.290622    2561 scope.go:117] "RemoveContainer" containerID="cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"
Jul 06 19:34:17.291012 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:34:17.290991    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253\": container with ID starting with cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253 not found: ID does not exist" containerID="cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"
Jul 06 19:34:17.291059 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.291022    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253"} err="failed to get container status \"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253\": rpc error: code = NotFound desc = could not find container \"cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253\": container with ID starting with cd85375641142a9f256e3a006d0a65b5ca408a56a552b850d303c3a0ce841253 not found: ID does not exist"
Jul 06 19:34:17.371498 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371450    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.371498 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371490    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.371498 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371502    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwfwt\" (UniqueName: \"kubernetes.io/projected/efd811cf-595b-4a23-b70a-112e961a81d9-kube-api-access-vwfwt\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.371749 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371512    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.371749 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371524    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efd811cf-595b-4a23-b70a-112e961a81d9-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.371749 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.371532    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efd811cf-595b-4a23-b70a-112e961a81d9-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:34:17.595620 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.595586    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:34:17.602576 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.602541    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/auth-test-model-06f95c-kserve-5697c65b6c-mmqbv"]
Jul 06 19:34:17.961558 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:17.961452    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd811cf-595b-4a23-b70a-112e961a81d9" path="/var/lib/kubelet/pods/efd811cf-595b-4a23-b70a-112e961a81d9/volumes"
Jul 06 19:34:54.191147 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.191106    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:34:54.193646 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.191498    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd811cf-595b-4a23-b70a-112e961a81d9" containerName="main"
Jul 06 19:34:54.193646 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.191511    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd811cf-595b-4a23-b70a-112e961a81d9" containerName="main"
Jul 06 19:34:54.193646 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.191603    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="efd811cf-595b-4a23-b70a-112e961a81d9" containerName="main"
Jul 06 19:34:54.194620 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.194604    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.200118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.200093    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"kube-root-ca.crt\""
Jul 06 19:34:54.201453 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.201013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"default-dockercfg-z7gx5\""
Jul 06 19:34:54.201453 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.201278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"sub-test-model-06f95c-kserve-self-signed-certs\""
Jul 06 19:34:54.201453 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.201330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"openshift-service-ca.crt\""
Jul 06 19:34:54.209280 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.209255    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:34:54.312073 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.312073 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.312302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2npx9\" (UniqueName: \"kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.312302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.312302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.312302 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.312273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413151 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413151 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413415 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413415 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413415 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413243    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2npx9\" (UniqueName: \"kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413415 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413728 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413694    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413876 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413743    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.413876 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.413781    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.415708 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.415691    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.415979 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.415961    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.425061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.425035    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2npx9\" (UniqueName: \"kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9\") pod \"sub-test-model-06f95c-kserve-55699fbf49-r85tr\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") " pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.503954 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.503855    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:34:54.645420 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:54.645391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:34:54.648061 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:34:54.648034    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0823de75_a4fc_4896_ae14_2332a90df4d9.slice/crio-66c111fed5aca33f7ca8081cf4277425ebd5c37668926b3b6a731fd129a371d0 WatchSource:0}: Error finding container 66c111fed5aca33f7ca8081cf4277425ebd5c37668926b3b6a731fd129a371d0: Status 404 returned error can't find the container with id 66c111fed5aca33f7ca8081cf4277425ebd5c37668926b3b6a731fd129a371d0
Jul 06 19:34:55.459492 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:55.459439    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" event={"ID":"0823de75-a4fc-4896-ae14-2332a90df4d9","Type":"ContainerStarted","Data":"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"}
Jul 06 19:34:55.459492 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:55.459500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" event={"ID":"0823de75-a4fc-4896-ae14-2332a90df4d9","Type":"ContainerStarted","Data":"66c111fed5aca33f7ca8081cf4277425ebd5c37668926b3b6a731fd129a371d0"}
Jul 06 19:34:55.478357 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:34:55.478307    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" podStartSLOduration=1.4782919429999999 podStartE2EDuration="1.478291943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:34:55.477390483 +0000 UTC m=+2398.063598476" watchObservedRunningTime="2026-07-06 19:34:55.478291943 +0000 UTC m=+2398.064499917"
Jul 06 19:35:04.504978 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:04.504937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:04.505374 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:04.505000    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:04.517817 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:04.517788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:05.516542 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:05.516514    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:43.780541 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:43.780441    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:35:58.800936 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:58.800862    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" podUID="0823de75-a4fc-4896-ae14-2332a90df4d9" containerName="main" containerID="cri-o://40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4" gracePeriod=45
Jul 06 19:35:59.065578 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.065555    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:59.091070 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091140    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091187    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091404 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091247    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091404 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091283    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2npx9\" (UniqueName: \"kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9\") pod \"0823de75-a4fc-4896-ae14-2332a90df4d9\" (UID: \"0823de75-a4fc-4896-ae14-2332a90df4d9\") "
Jul 06 19:35:59.091551 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091290    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home" (OuterVolumeSpecName: "home") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:35:59.091614 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091418    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:35:59.091838 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091810    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:35:59.091838 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091829    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.091969 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.091854    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.093353 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.093332    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm" (OuterVolumeSpecName: "dshm") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:35:59.093590 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.093568    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:35:59.093925 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.093902    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9" (OuterVolumeSpecName: "kube-api-access-2npx9") pod "0823de75-a4fc-4896-ae14-2332a90df4d9" (UID: "0823de75-a4fc-4896-ae14-2332a90df4d9"). InnerVolumeSpecName "kube-api-access-2npx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:35:59.192998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.192957    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.192998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.192988    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0823de75-a4fc-4896-ae14-2332a90df4d9-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.192998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.192998    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0823de75-a4fc-4896-ae14-2332a90df4d9-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.192998 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.193007    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2npx9\" (UniqueName: \"kubernetes.io/projected/0823de75-a4fc-4896-ae14-2332a90df4d9-kube-api-access-2npx9\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:35:59.746086 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.746052    2561 generic.go:356] "Generic (PLEG): container finished" podID="0823de75-a4fc-4896-ae14-2332a90df4d9" containerID="40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4" exitCode=0
Jul 06 19:35:59.746267 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.746095    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"
Jul 06 19:35:59.746267 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.746134    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" event={"ID":"0823de75-a4fc-4896-ae14-2332a90df4d9","Type":"ContainerDied","Data":"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"}
Jul 06 19:35:59.746267 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.746178    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr" event={"ID":"0823de75-a4fc-4896-ae14-2332a90df4d9","Type":"ContainerDied","Data":"66c111fed5aca33f7ca8081cf4277425ebd5c37668926b3b6a731fd129a371d0"}
Jul 06 19:35:59.746267 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.746195    2561 scope.go:117] "RemoveContainer" containerID="40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"
Jul 06 19:35:59.755159 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.755131    2561 scope.go:117] "RemoveContainer" containerID="40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"
Jul 06 19:35:59.755424 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:35:59.755405    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4\": container with ID starting with 40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4 not found: ID does not exist" containerID="40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"
Jul 06 19:35:59.755522 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.755431    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4"} err="failed to get container status \"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4\": rpc error: code = NotFound desc = could not find container \"40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4\": container with ID starting with 40ef3f7cfbd47842749fade54bee90e78a3208ef77487f7d7cce4f5848c892a4 not found: ID does not exist"
Jul 06 19:35:59.767244 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.767217    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:35:59.771194 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.771161    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/sub-test-model-06f95c-kserve-55699fbf49-r85tr"]
Jul 06 19:35:59.962333 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:35:59.962295    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0823de75-a4fc-4896-ae14-2332a90df4d9" path="/var/lib/kubelet/pods/0823de75-a4fc-4896-ae14-2332a90df4d9/volumes"
Jul 06 19:36:25.906869 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.906830    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:36:25.907275 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.907183    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0823de75-a4fc-4896-ae14-2332a90df4d9" containerName="main"
Jul 06 19:36:25.907275 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.907194    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0823de75-a4fc-4896-ae14-2332a90df4d9" containerName="main"
Jul 06 19:36:25.907275 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.907275    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0823de75-a4fc-4896-ae14-2332a90df4d9" containerName="main"
Jul 06 19:36:25.910314 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.910296    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:25.913137 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.913113    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"openshift-service-ca.crt\""
Jul 06 19:36:25.914208 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.914191    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"rate-test-model-06f95c-kserve-self-signed-certs\""
Jul 06 19:36:25.914296 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.914210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"default-dockercfg-z7gx5\""
Jul 06 19:36:25.914296 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.914214    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-06f95c\"/\"kube-root-ca.crt\""
Jul 06 19:36:25.919577 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:25.919555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:36:26.040106 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.040106 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.040324 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595bn\" (UniqueName: \"kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.040324 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.040324 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.040324 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.040311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.140828 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.140782    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595bn\" (UniqueName: \"kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141031 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.140839    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141031 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.140857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141031 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141017    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141303 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141303 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141413 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141366    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141486 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141427    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.141486 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.141431    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.143495 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.143451    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.143608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.143545    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.149799 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.149774    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-595bn\" (UniqueName: \"kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn\") pod \"rate-test-model-06f95c-kserve-9bbc99fff-gst2c\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") " pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.220708 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.220617    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:26.364411 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.364381    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:36:26.367210 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:36:26.367169    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568fdb4d_716e_4579_8ab2_b84a39e927fe.slice/crio-f23d45e642c7bf3c9128fe855f6e12e52d52c904ab5e125ce3857a66da8e08d0 WatchSource:0}: Error finding container f23d45e642c7bf3c9128fe855f6e12e52d52c904ab5e125ce3857a66da8e08d0: Status 404 returned error can't find the container with id f23d45e642c7bf3c9128fe855f6e12e52d52c904ab5e125ce3857a66da8e08d0
Jul 06 19:36:26.871830 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.871790    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" event={"ID":"568fdb4d-716e-4579-8ab2-b84a39e927fe","Type":"ContainerStarted","Data":"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"}
Jul 06 19:36:26.871830 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.871833    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" event={"ID":"568fdb4d-716e-4579-8ab2-b84a39e927fe","Type":"ContainerStarted","Data":"f23d45e642c7bf3c9128fe855f6e12e52d52c904ab5e125ce3857a66da8e08d0"}
Jul 06 19:36:26.892156 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:26.892100    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" podStartSLOduration=1.892083269 podStartE2EDuration="1.892083269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:36:26.890168994 +0000 UTC m=+2489.476376968" watchObservedRunningTime="2026-07-06 19:36:26.892083269 +0000 UTC m=+2489.478291243"
Jul 06 19:36:36.221623 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:36.221580    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:36.222209 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:36.221642    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:36.237117 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:36.237084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:36:36.928163 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:36:36.928124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:37:11.285524 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:11.285399    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:37:13.201018 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.200982    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:37:13.201438 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.201209    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" podUID="f3226313-f8df-4847-8fc4-2942bfd92484" containerName="maas-api" containerID="cri-o://09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4" gracePeriod=30
Jul 06 19:37:13.455044 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.454968    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:37:13.559175 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.559131    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk5tx\" (UniqueName: \"kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx\") pod \"f3226313-f8df-4847-8fc4-2942bfd92484\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") "
Jul 06 19:37:13.559378 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.559211    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca\") pod \"f3226313-f8df-4847-8fc4-2942bfd92484\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") "
Jul 06 19:37:13.559378 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.559320    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls\") pod \"f3226313-f8df-4847-8fc4-2942bfd92484\" (UID: \"f3226313-f8df-4847-8fc4-2942bfd92484\") "
Jul 06 19:37:13.559667 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.559641    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3226313-f8df-4847-8fc4-2942bfd92484" (UID: "f3226313-f8df-4847-8fc4-2942bfd92484"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:37:13.561532 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.561497    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx" (OuterVolumeSpecName: "kube-api-access-dk5tx") pod "f3226313-f8df-4847-8fc4-2942bfd92484" (UID: "f3226313-f8df-4847-8fc4-2942bfd92484"). InnerVolumeSpecName "kube-api-access-dk5tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:37:13.561532 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.561496    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3226313-f8df-4847-8fc4-2942bfd92484" (UID: "f3226313-f8df-4847-8fc4-2942bfd92484"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:37:13.660161 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.660118    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3226313-f8df-4847-8fc4-2942bfd92484-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:13.660161 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.660151    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5tx\" (UniqueName: \"kubernetes.io/projected/f3226313-f8df-4847-8fc4-2942bfd92484-kube-api-access-dk5tx\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:13.660161 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:13.660162    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3226313-f8df-4847-8fc4-2942bfd92484-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:14.084188 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.084148    2561 generic.go:356] "Generic (PLEG): container finished" podID="f3226313-f8df-4847-8fc4-2942bfd92484" containerID="09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4" exitCode=0
Jul 06 19:37:14.084188 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.084192    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"
Jul 06 19:37:14.084488 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.084219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" event={"ID":"f3226313-f8df-4847-8fc4-2942bfd92484","Type":"ContainerDied","Data":"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"}
Jul 06 19:37:14.084488 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.084272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2" event={"ID":"f3226313-f8df-4847-8fc4-2942bfd92484","Type":"ContainerDied","Data":"e94ac6b972362a9fc3024164069ee36ac2caefcb9838f251a2b5eb90130d6e20"}
Jul 06 19:37:14.084488 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.084299    2561 scope.go:117] "RemoveContainer" containerID="09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"
Jul 06 19:37:14.093256 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.093236    2561 scope.go:117] "RemoveContainer" containerID="09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"
Jul 06 19:37:14.093561 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:37:14.093538    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4\": container with ID starting with 09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4 not found: ID does not exist" containerID="09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"
Jul 06 19:37:14.093645 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.093569    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4"} err="failed to get container status \"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4\": rpc error: code = NotFound desc = could not find container \"09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4\": container with ID starting with 09be44221fa408a84bc6568b679909de640ed2ea5ee4ee0c75e873539b8963a4 not found: ID does not exist"
Jul 06 19:37:14.101972 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.101945    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:37:14.108135 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:14.108109    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-3550d9-9568c77dd-545l2"]
Jul 06 19:37:15.962085 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:15.962047    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3226313-f8df-4847-8fc4-2942bfd92484" path="/var/lib/kubelet/pods/f3226313-f8df-4847-8fc4-2942bfd92484/volumes"
Jul 06 19:37:26.305964 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.305918    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" podUID="568fdb4d-716e-4579-8ab2-b84a39e927fe" containerName="main" containerID="cri-o://f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce" gracePeriod=45
Jul 06 19:37:26.568894 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.568870    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:37:26.680963 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.680908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.680990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-595bn\" (UniqueName: \"kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681037    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681100    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681182 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681172    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache\") pod \"568fdb4d-716e-4579-8ab2-b84a39e927fe\" (UID: \"568fdb4d-716e-4579-8ab2-b84a39e927fe\") "
Jul 06 19:37:26.681490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681394    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:37:26.681490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681424    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home" (OuterVolumeSpecName: "home") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:37:26.681490 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.681435    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:37:26.683500 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.683446    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm" (OuterVolumeSpecName: "dshm") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 19:37:26.683500 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.683440    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn" (OuterVolumeSpecName: "kube-api-access-595bn") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "kube-api-access-595bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:37:26.683500 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.683445    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "568fdb4d-716e-4579-8ab2-b84a39e927fe" (UID: "568fdb4d-716e-4579-8ab2-b84a39e927fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:37:26.782283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782238    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-model-cache\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:26.782283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782273    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/568fdb4d-716e-4579-8ab2-b84a39e927fe-tls-certs\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:26.782283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782282    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-595bn\" (UniqueName: \"kubernetes.io/projected/568fdb4d-716e-4579-8ab2-b84a39e927fe-kube-api-access-595bn\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:26.782283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782291    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-dshm\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:26.782608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782301    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-home\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:26.782608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:26.782308    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/568fdb4d-716e-4579-8ab2-b84a39e927fe-tmp-dir\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:27.142283 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.142253    2561 generic.go:356] "Generic (PLEG): container finished" podID="568fdb4d-716e-4579-8ab2-b84a39e927fe" containerID="f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce" exitCode=0
Jul 06 19:37:27.142454 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.142295    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"
Jul 06 19:37:27.142454 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.142328    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" event={"ID":"568fdb4d-716e-4579-8ab2-b84a39e927fe","Type":"ContainerDied","Data":"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"}
Jul 06 19:37:27.142454 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.142375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c" event={"ID":"568fdb4d-716e-4579-8ab2-b84a39e927fe","Type":"ContainerDied","Data":"f23d45e642c7bf3c9128fe855f6e12e52d52c904ab5e125ce3857a66da8e08d0"}
Jul 06 19:37:27.142454 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.142393    2561 scope.go:117] "RemoveContainer" containerID="f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"
Jul 06 19:37:27.151746 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.151725    2561 scope.go:117] "RemoveContainer" containerID="f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"
Jul 06 19:37:27.152013 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:37:27.151994    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce\": container with ID starting with f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce not found: ID does not exist" containerID="f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"
Jul 06 19:37:27.152061 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.152028    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce"} err="failed to get container status \"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce\": rpc error: code = NotFound desc = could not find container \"f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce\": container with ID starting with f5efeb895daba810f50821854b80f8954931789c6255cadb436eaef908c913ce not found: ID does not exist"
Jul 06 19:37:27.177680 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.177645    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:37:27.180712 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.180685    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-06f95c/rate-test-model-06f95c-kserve-9bbc99fff-gst2c"]
Jul 06 19:37:27.964111 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:27.964077    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="568fdb4d-716e-4579-8ab2-b84a39e927fe" path="/var/lib/kubelet/pods/568fdb4d-716e-4579-8ab2-b84a39e927fe/volumes"
Jul 06 19:37:56.311343 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.311305    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:37:56.311968 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.311549    2561 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" podUID="c8cc6167-1353-49e2-af47-bab82ed6c112" containerName="maas-api" containerID="cri-o://2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7" gracePeriod=30
Jul 06 19:37:56.565838 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.565773    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:37:56.639837 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.639797    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca\") pod \"c8cc6167-1353-49e2-af47-bab82ed6c112\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") "
Jul 06 19:37:56.639837 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.639844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls\") pod \"c8cc6167-1353-49e2-af47-bab82ed6c112\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") "
Jul 06 19:37:56.640107 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.639891    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sc6z\" (UniqueName: \"kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z\") pod \"c8cc6167-1353-49e2-af47-bab82ed6c112\" (UID: \"c8cc6167-1353-49e2-af47-bab82ed6c112\") "
Jul 06 19:37:56.640247 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.640210    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8cc6167-1353-49e2-af47-bab82ed6c112" (UID: "c8cc6167-1353-49e2-af47-bab82ed6c112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 19:37:56.642264 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.642235    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z" (OuterVolumeSpecName: "kube-api-access-7sc6z") pod "c8cc6167-1353-49e2-af47-bab82ed6c112" (UID: "c8cc6167-1353-49e2-af47-bab82ed6c112"). InnerVolumeSpecName "kube-api-access-7sc6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 19:37:56.642351 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.642262    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8cc6167-1353-49e2-af47-bab82ed6c112" (UID: "c8cc6167-1353-49e2-af47-bab82ed6c112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 19:37:56.741033 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.740992    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8cc6167-1353-49e2-af47-bab82ed6c112-openshift-service-ca\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:56.741033 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.741026    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8cc6167-1353-49e2-af47-bab82ed6c112-maas-api-tls\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:56.741033 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:56.741035    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sc6z\" (UniqueName: \"kubernetes.io/projected/c8cc6167-1353-49e2-af47-bab82ed6c112-kube-api-access-7sc6z\") on node \"ip-10-0-138-123.ec2.internal\" DevicePath \"\""
Jul 06 19:37:57.281393 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.281361    2561 generic.go:356] "Generic (PLEG): container finished" podID="c8cc6167-1353-49e2-af47-bab82ed6c112" containerID="2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7" exitCode=0
Jul 06 19:37:57.281608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.281409    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"
Jul 06 19:37:57.281608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.281454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" event={"ID":"c8cc6167-1353-49e2-af47-bab82ed6c112","Type":"ContainerDied","Data":"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"}
Jul 06 19:37:57.281608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.281531    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr" event={"ID":"c8cc6167-1353-49e2-af47-bab82ed6c112","Type":"ContainerDied","Data":"ec680a282d1eba97e26085db36bd05b82c644b40dbab2a2c45d9019cb558a879"}
Jul 06 19:37:57.281608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.281560    2561 scope.go:117] "RemoveContainer" containerID="2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"
Jul 06 19:37:57.290785 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.290761    2561 scope.go:117] "RemoveContainer" containerID="2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"
Jul 06 19:37:57.291082 ip-10-0-138-123 kubenswrapper[2561]: E0706 19:37:57.291062    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7\": container with ID starting with 2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7 not found: ID does not exist" containerID="2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"
Jul 06 19:37:57.291140 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.291092    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7"} err="failed to get container status \"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7\": rpc error: code = NotFound desc = could not find container \"2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7\": container with ID starting with 2a85ff5107795edd05fded7f16dcd5823cc951f9f57a019fe9804189448b73a7 not found: ID does not exist"
Jul 06 19:37:57.849776 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:57.849738    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:37:58.116868 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:58.116777    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-06f95c-8685698754-97cqr"]
Jul 06 19:37:59.962749 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:37:59.962716    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8cc6167-1353-49e2-af47-bab82ed6c112" path="/var/lib/kubelet/pods/c8cc6167-1353-49e2-af47-bab82ed6c112/volumes"
Jul 06 19:38:00.499512 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:00.499436    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7bf8d74cb9-tfnq8" podUID="a6aab718-837a-4515-80a6-7379f5b5b0f8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 06 19:38:11.363568 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:11.363534    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q52mx_5b572ebf-05e3-48a2-b944-580b6571dc04/manager/0.log"
Jul 06 19:38:11.492276 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:11.492238    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68cb9b94f4-xzgkp_afda3049-3c2b-4af3-9800-e9688e7d73d9/maas-api/0.log"
Jul 06 19:38:11.864309 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:11.864278    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bf8d74cb9-tfnq8_a6aab718-837a-4515-80a6-7379f5b5b0f8/manager/0.log"
Jul 06 19:38:12.517924 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:12.517897    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546c6c6d78-zdx6s_a4162355-478e-47b2-b65b-e9c652a13074/copy-manifests/0.log"
Jul 06 19:38:12.525968 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:12.525940    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546c6c6d78-zdx6s_a4162355-478e-47b2-b65b-e9c652a13074/manager/0.log"
Jul 06 19:38:12.641826 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:12.641793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-gbnsl_a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed/postgres/0.log"
Jul 06 19:38:14.381861 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:14.381832    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cbfm8_9a89d790-aa8a-47e6-89f8-507605ad13ba/kuadrant-console-plugin/0.log"
Jul 06 19:38:14.505111 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:14.505076    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7m9s4_c2a51fec-a938-479c-972a-108694e4d5c7/registry-server/0.log"
Jul 06 19:38:14.632631 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:14.632545    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5wqw8_527770fd-d5f9-4520-8fb4-bfbcbd28b70d/manager/0.log"
Jul 06 19:38:15.864970 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:15.864942    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-thrsd_f5f894d5-6cd5-4bbf-99eb-82e4d9d91458/payload-pre-processing/0.log"
Jul 06 19:38:16.483119 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:16.483090    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-njzhc_43d79110-14d4-4e6d-bb8f-ff1ffeb7e76b/main/0.log"
Jul 06 19:38:16.599193 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:16.599156    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k94d2_aadce368-26e4-4c19-895d-b4c05237d34b/main/0.log"
Jul 06 19:38:16.834210 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:16.834163    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbnjwj5_be7d83fc-0f48-497f-856e-c3f95d2bfb73/main/0.log"
Jul 06 19:38:24.157327 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:24.157221    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-56wjc_093150fe-2f19-46de-a8e3-714e7ee0779c/global-pull-secret-syncer/0.log"
Jul 06 19:38:24.382910 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:24.382878    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zch6z_8485ac65-9021-4ac1-97fb-cdc1bc397845/konnectivity-agent/0.log"
Jul 06 19:38:24.429856 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:24.429774    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-123.ec2.internal_a7389aae70d56334e642e67fe3222430/haproxy/0.log"
Jul 06 19:38:30.054611 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:30.054583    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cbfm8_9a89d790-aa8a-47e6-89f8-507605ad13ba/kuadrant-console-plugin/0.log"
Jul 06 19:38:30.099922 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:30.099892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7m9s4_c2a51fec-a938-479c-972a-108694e4d5c7/registry-server/0.log"
Jul 06 19:38:30.210883 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:30.210847    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-5wqw8_527770fd-d5f9-4520-8fb4-bfbcbd28b70d/manager/0.log"
Jul 06 19:38:32.385608 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:32.385576    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2kv4_27b6e62d-d216-4718-8d93-778111bfd810/node-exporter/0.log"
Jul 06 19:38:32.414338 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:32.414317    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2kv4_27b6e62d-d216-4718-8d93-778111bfd810/kube-rbac-proxy/0.log"
Jul 06 19:38:32.444785 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:32.444756    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2kv4_27b6e62d-d216-4718-8d93-778111bfd810/init-textfile/0.log"
Jul 06 19:38:34.188524 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188488    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"]
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188824    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3226313-f8df-4847-8fc4-2942bfd92484" containerName="maas-api"
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188835    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3226313-f8df-4847-8fc4-2942bfd92484" containerName="maas-api"
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188850    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cc6167-1353-49e2-af47-bab82ed6c112" containerName="maas-api"
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188857    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cc6167-1353-49e2-af47-bab82ed6c112" containerName="maas-api"
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188863    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="568fdb4d-716e-4579-8ab2-b84a39e927fe" containerName="main"
Jul 06 19:38:34.188906 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188870    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="568fdb4d-716e-4579-8ab2-b84a39e927fe" containerName="main"
Jul 06 19:38:34.189098 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188924    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3226313-f8df-4847-8fc4-2942bfd92484" containerName="maas-api"
Jul 06 19:38:34.189098 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188932    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cc6167-1353-49e2-af47-bab82ed6c112" containerName="maas-api"
Jul 06 19:38:34.189098 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.188938    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="568fdb4d-716e-4579-8ab2-b84a39e927fe" containerName="main"
Jul 06 19:38:34.191837 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.191813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.194425 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.194406    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-446hm\"/\"kube-root-ca.crt\""
Jul 06 19:38:34.194569 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.194485    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-446hm\"/\"openshift-service-ca.crt\""
Jul 06 19:38:34.195559 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.195537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-446hm\"/\"default-dockercfg-vqxjx\""
Jul 06 19:38:34.201617 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.201591    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"]
Jul 06 19:38:34.269040 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.268994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-podres\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.269237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.269076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbbb\" (UniqueName: \"kubernetes.io/projected/0940eb9f-841d-4394-883e-0000549ae9c3-kube-api-access-bvbbb\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.269237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.269123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-sys\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.269237 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.269146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-lib-modules\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.269387 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.269284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-proc\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.369909 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.369863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-sys\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370064 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.369918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-lib-modules\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370064 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.369998    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-sys\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370064 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370014    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-proc\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370064 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370042    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-podres\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370064 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbbb\" (UniqueName: \"kubernetes.io/projected/0940eb9f-841d-4394-883e-0000549ae9c3-kube-api-access-bvbbb\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370084    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-proc\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370134    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-lib-modules\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.370251 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.370184    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0940eb9f-841d-4394-883e-0000549ae9c3-podres\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.376792 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.376764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-5hnm4_6fccd601-0088-4800-864d-0c1b7bf65c64/networking-console-plugin/0.log"
Jul 06 19:38:34.379012 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.378990    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbbb\" (UniqueName: \"kubernetes.io/projected/0940eb9f-841d-4394-883e-0000549ae9c3-kube-api-access-bvbbb\") pod \"perf-node-gather-daemonset-b6tg8\" (UID: \"0940eb9f-841d-4394-883e-0000549ae9c3\") " pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.503131 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.503041    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:34.641027 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:34.640940    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"]
Jul 06 19:38:34.644854 ip-10-0-138-123 kubenswrapper[2561]: W0706 19:38:34.644815    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0940eb9f_841d_4394_883e_0000549ae9c3.slice/crio-0df62bf1e5d1377158b41140d9661b4d9b4bd3c8d39f08367bd34b83e4cccd67 WatchSource:0}: Error finding container 0df62bf1e5d1377158b41140d9661b4d9b4bd3c8d39f08367bd34b83e4cccd67: Status 404 returned error can't find the container with id 0df62bf1e5d1377158b41140d9661b4d9b4bd3c8d39f08367bd34b83e4cccd67
Jul 06 19:38:35.451363 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:35.451324    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8" event={"ID":"0940eb9f-841d-4394-883e-0000549ae9c3","Type":"ContainerStarted","Data":"d920da39c93c46bdc405ca9774ccd71a0bbe3c5eadbf2bb0338ff93394a991e7"}
Jul 06 19:38:35.451865 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:35.451378    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8" event={"ID":"0940eb9f-841d-4394-883e-0000549ae9c3","Type":"ContainerStarted","Data":"0df62bf1e5d1377158b41140d9661b4d9b4bd3c8d39f08367bd34b83e4cccd67"}
Jul 06 19:38:35.451865 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:35.451407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:35.469446 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:35.469394    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8" podStartSLOduration=1.469377145 podStartE2EDuration="1.469377145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:35.468064962 +0000 UTC m=+2618.054272936" watchObservedRunningTime="2026-07-06 19:38:35.469377145 +0000 UTC m=+2618.055585119"
Jul 06 19:38:37.144786 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:37.144756    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-596vd_ea86a2af-8ed1-43ea-b256-636f7563fb0c/dns/0.log"
Jul 06 19:38:37.180112 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:37.180081    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-596vd_ea86a2af-8ed1-43ea-b256-636f7563fb0c/kube-rbac-proxy/0.log"
Jul 06 19:38:37.434492 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:37.434392    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q95qr_cc8e9d9f-43be-4e26-b5f6-04e8177a4d47/dns-node-resolver/0.log"
Jul 06 19:38:38.048025 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:38.047987    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7856c7b44-tnrlb_485fee5f-4f28-4c04-bdcf-bf4d2905334e/registry/0.log"
Jul 06 19:38:38.075474 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:38.075431    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j4jpp_fcc43497-d4f3-4732-b4ab-a3c351f6b64e/node-ca/0.log"
Jul 06 19:38:39.551084 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:39.551053    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-thrsd_f5f894d5-6cd5-4bbf-99eb-82e4d9d91458/payload-pre-processing/0.log"
Jul 06 19:38:40.398141 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:40.398107    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ln5nv_3c3e2b89-9407-4bed-b547-c2a6b1681311/serve-healthcheck-canary/0.log"
Jul 06 19:38:41.084399 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:41.084369    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz46j_429a4284-f11f-4db9-a202-4e6691a3961b/kube-rbac-proxy/0.log"
Jul 06 19:38:41.109504 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:41.109455    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz46j_429a4284-f11f-4db9-a202-4e6691a3961b/exporter/0.log"
Jul 06 19:38:41.136399 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:41.136366    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pz46j_429a4284-f11f-4db9-a202-4e6691a3961b/extractor/0.log"
Jul 06 19:38:41.465993 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:41.465909    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-446hm/perf-node-gather-daemonset-b6tg8"
Jul 06 19:38:43.614118 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:43.614083    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q52mx_5b572ebf-05e3-48a2-b944-580b6571dc04/manager/0.log"
Jul 06 19:38:43.654866 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:43.654834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-68cb9b94f4-xzgkp_afda3049-3c2b-4af3-9800-e9688e7d73d9/maas-api/0.log"
Jul 06 19:38:43.819966 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:43.819930    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bf8d74cb9-tfnq8_a6aab718-837a-4515-80a6-7379f5b5b0f8/manager/0.log"
Jul 06 19:38:44.216203 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:44.216176    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546c6c6d78-zdx6s_a4162355-478e-47b2-b65b-e9c652a13074/manager/0.log"
Jul 06 19:38:44.240419 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:44.240395    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-546c6c6d78-zdx6s_a4162355-478e-47b2-b65b-e9c652a13074/copy-manifests/0.log"
Jul 06 19:38:44.275343 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:44.275313    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-gbnsl_a5f90f3d-2299-46fa-bc93-e9f6ee4aa1ed/postgres/0.log"
Jul 06 19:38:45.062142 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:45.062111    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-thrsd_f5f894d5-6cd5-4bbf-99eb-82e4d9d91458/payload-pre-processing/0.log"
Jul 06 19:38:45.703851 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:45.703766    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d5c698c7c-t6jpf_7574516d-6feb-423f-8143-0609aaceed05/manager/0.log"
Jul 06 19:38:46.678528 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:46.678502    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-thrsd_f5f894d5-6cd5-4bbf-99eb-82e4d9d91458/payload-pre-processing/0.log"
Jul 06 19:38:50.628437 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:50.628410    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bxdql_be3fbda2-88be-41df-ac72-f75e40e5f29e/migrator/0.log"
Jul 06 19:38:50.652235 ip-10-0-138-123 kubenswrapper[2561]: I0706 19:38:50.652204    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-bxdql_be3fbda2-88be-41df-ac72-f75e40e5f29e/graceful-termination/0.log"