Jul 17 18:44:34.108607 ip-10-0-138-140 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 18:44:34.563415 ip-10-0-138-140 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:44:34.563415 ip-10-0-138-140 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 18:44:34.563415 ip-10-0-138-140 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:44:34.563415 ip-10-0-138-140 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 18:44:34.563415 ip-10-0-138-140 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 18:44:34.564575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.564413    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 18:44:34.570339 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570311    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.570339 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570339    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570343    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570346    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570351    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570356    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570359    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570362    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570365    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570368    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570371    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570374    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570377    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570380    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570382    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570385    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570387    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570391    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570394    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570396    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570399    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.570408 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570401    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570404    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570406    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570409    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570412    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570414    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570417    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570421    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570424    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570427    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570429    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570432    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570434    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570437    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570439    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570442    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570444    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570446    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570449    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.570882 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570452    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570454    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570457    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570460    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570462    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570464    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570467    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570469    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570472    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570474    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570476    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570479    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570481    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570483    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570486    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570490    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570492    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570495    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570497    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.571354 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570500    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570502    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570505    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570507    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570510    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570513    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570516    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570518    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570521    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570523    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570526    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570529    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570533    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570535    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570538    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570541    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570544    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570546    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570548    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570551    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.571845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570553    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570556    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570560    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570562    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570566    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570570    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570573    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570575    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570580    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570582    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570585    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570588    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570591    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570594    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570596    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570599    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570601    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570604    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570608    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.572355 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.570611    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571337    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571345    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571349    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571353    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571356    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571361    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571364    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571367    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571370    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571373    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571375    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571378    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571381    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571383    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571385    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571388    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571391    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571393    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.572823 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571396    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571399    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571402    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571404    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571407    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571410    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571412    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571415    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571417    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571419    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571422    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571424    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571426    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571429    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571431    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571433    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571436    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571438    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571441    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571443    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.573356 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571446    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571449    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571451    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571453    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571456    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571458    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571461    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571463    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571465    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571468    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571471    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571473    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571476    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571478    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571481    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571484    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571487    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571489    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571492    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571494    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.573885 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571497    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571499    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571501    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571504    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571506    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571508    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571511    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571514    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571517    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571520    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571523    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571526    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571528    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571531    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571534    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571537    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571539    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571541    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571544    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.574373 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571546    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571548    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571551    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571554    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571556    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571558    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571561    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571563    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571566    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571568    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571571    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571574    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571577    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571579    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571582    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571584    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571586    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571589    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571591    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.574836 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571595    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571598    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571601    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571961    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571966    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571969    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571973    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571977    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571980    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571984    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571988    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571991    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571994    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571997    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.571999    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572002    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572004    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572007    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572009    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.575288 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572011    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572015    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572017    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572020    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572022    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572025    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572027    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572030    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572032    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572034    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572037    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572039    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572042    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572044    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572046    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572049    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572051    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572054    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572056    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572059    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.575761 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572064    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572070    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572072    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572075    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572077    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572080    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572083    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572086    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572088    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572091    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572093    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572096    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572098    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572101    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572103    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572106    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572108    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572111    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572113    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572116    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.576237 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572118    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572121    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572123    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572126    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572128    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572131    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572133    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572136    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572138    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572140    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572143    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572145    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572151    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572153    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572158    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572162    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572165    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572168    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572170    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.576743 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572173    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572175    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572178    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572180    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572183    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572185    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572187    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572190    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572192    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572195    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572197    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572200    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572202    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572205    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572207    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572209    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572212    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572214    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572218    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572221    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.577307 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572224    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572227    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572229    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572231    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572305    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572333    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572340    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572345    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572350    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572354    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572359    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572363    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572366    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572369    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572373    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572377    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572380    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572382    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572385    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572388    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572391    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572394    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572398    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572401    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 18:44:34.577842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572404    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572408    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572411    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572415    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572418    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572421    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572424    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572427    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572430    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572433    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572436    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572438    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572443    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572447    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572449    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572452    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572456    2558 flags.go:64] FLAG: --enable-server="true"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572460    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572464    2558 flags.go:64] FLAG: --event-burst="100"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572467    2558 flags.go:64] FLAG: --event-qps="50"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572470    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572472    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572475    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572479    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 18:44:34.578442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572482    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572485    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572488    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572490    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572493    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572496    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572499    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572502    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572504    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572507    2558 flags.go:64] FLAG: --feature-gates=""
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572511    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572514    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572517    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572520    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572523    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572526    2558 flags.go:64] FLAG: --help="false"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572528    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572531    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572534    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572537    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572540    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572545    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572547    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 18:44:34.579013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572550    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572557    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572560    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572564    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572567    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572569    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572572    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572575    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572578    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572581    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572584    2558 flags.go:64] FLAG: --lock-file=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572587    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572590    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572593    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572598    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572601    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572604    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572606    2558 flags.go:64] FLAG: --logging-format="text"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572609    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572612    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572615    2558 flags.go:64] FLAG: --manifest-url=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572618    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572622    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572625    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572628    2558 flags.go:64] FLAG: --max-pods="110"
Jul 17 18:44:34.579650 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572631    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572634    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572637    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572640    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572643    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572647    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572650    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572657    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572661    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572664    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572667    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572670    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572676    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572679    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572681    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572684    2558 flags.go:64] FLAG: --port="10250"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572687    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572690    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d0ccd48ba2a23a6f"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572693    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572696    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572699    2558 flags.go:64] FLAG: --register-node="true"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572701    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572705    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572707    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 17 18:44:34.580286 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572710    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572713    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572716    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572719    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572722    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572725    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572728    2558 flags.go:64] FLAG: --runonce="false"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572730    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572733    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572736    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572739    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572741    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572744    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572747    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572751    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572753    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572758    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572760    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572763    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572766    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572769    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572772    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572777    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572780    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572783    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 18:44:34.580894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572787    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572790    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572792    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572795    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572798    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572801    2558 flags.go:64] FLAG: --v="2"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572804    2558 flags.go:64] FLAG: --version="false"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572808    2558 flags.go:64] FLAG: --vmodule=""
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572811    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.572814    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572935    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572939    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572942    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572945    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572947    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572950    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572952    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572955    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572957    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572960    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572962    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572965    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.581511 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572968    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572972    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572975    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572977    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572980    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572982    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572985    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572988    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572990    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572993    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572995    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.572998    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573000    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573003    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573005    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573008    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573010    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573013    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573015    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.582032 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573017    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573020    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573022    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573024    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573027    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573029    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573031    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573034    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573037    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573039    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573042    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573045    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573047    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573049    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573054    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573057    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573061    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573064    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573067    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573069    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.582536 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573072    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573074    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573078    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573082    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573084    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573087    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573090    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573094    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573096    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573099    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573102    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573104    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573107    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573110    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573112    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573115    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573118    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573120    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573123    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.583062 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573125    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573128    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573130    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573133    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573135    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573137    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573140    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573144    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573146    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573149    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573151    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573154    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573156    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573158    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573161    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573164    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573166    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573168    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573171    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573173    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.583549 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573175    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573178    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573180    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573183    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573185    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573187    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573190    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573192    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.573194    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.573982    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.581230    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.581246    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.581266    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.581276    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581348    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.584176 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581353    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581357    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581360    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581363    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581366    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581369    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581372    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581376    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581380    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581383    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581386    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581389    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581391    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581394    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581398    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581400    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581403    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581406    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581408    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.584658 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581412    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581415    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581418    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581421    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581424    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581427    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581430    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581433    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581436    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581439    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581442    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581445    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581447    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581450    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581452    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581455    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581458    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581460    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581463    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581465    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.585137 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581468    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581470    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581472    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581475    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581478    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581480    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581483    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581485    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581488    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581490    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581492    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581495    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581497    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581500    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581502    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581504    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581507    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581510    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581512    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.585666 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581515    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581518    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581521    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581526    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581530    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581533    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581535    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581537    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581540    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581543    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581545    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581548    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581550    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581553    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581556    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581558    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581560    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581563    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581565    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581568    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.586144 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581570    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581573    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581575    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581577    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581580    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581582    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581585    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581588    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581590    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581592    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581595    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581597    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581600    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581603    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581605    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581607    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581610    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581613    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581615    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.586683 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581618    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.581623    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581734    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581739    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581742    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581745    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581748    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581750    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581753    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581756    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581760    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581763    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581766    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581769    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 18:44:34.587141 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581771    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581774    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581776    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581778    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581781    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581783    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581786    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581788    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581790    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581793    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581796    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581798    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581801    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581803    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581805    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581809    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581813    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581816    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581818    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 18:44:34.587532 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581820    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581823    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581825    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581827    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581830    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581832    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581834    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581837    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581839    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581842    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581844    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581846    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581849    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581852    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581854    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581857    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581859    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581861    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581864    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 18:44:34.587990 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581866    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581868    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581871    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581873    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581876    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581878    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581880    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581883    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581885    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581888    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581890    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581898    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581901    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581903    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581906    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581908    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581911    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581913    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581916    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581918    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 18:44:34.588473 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581921    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581923    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581926    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581928    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581931    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581933    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581936    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581938    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581940    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581943    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581946    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581948    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581951    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581953    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581955    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581958    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581960    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581963    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581965    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581967    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 18:44:34.588958 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581970    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581973    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581977    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581979    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581982    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581985    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581991    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581993    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:34.581996    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.582001    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.582100    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.587186    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.588906    2558 server.go:1019] "Starting client certificate rotation"
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.588998    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:44:34.589474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.589050    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 18:44:34.591756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.591741    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:44:34.592949 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.592926    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 18:44:34.600707 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.600692    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 17 18:44:34.603134 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.603120    2558 log.go:25] "Validated CRI v1 image API"
Jul 17 18:44:34.603920 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.603907    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 18:44:34.608947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.608924    2558 fs.go:135] Filesystem UUIDs: map[78f4e71e-cccf-4108-a7a0-b1f2337dc43d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 928c68d8-185d-4144-b2fe-69d907181e37:/dev/nvme0n1p4]
Jul 17 18:44:34.608999 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.608947    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 18:44:34.615068 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.614952    2558 manager.go:216] Machine: {Timestamp:2026-07-17 18:44:34.613557274 +0000 UTC m=+0.375987545 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3063934 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ad7dbb9ca744e58302ba077bb1f5d SystemUUID:ec2ad7db-b9ca-744e-5830-2ba077bb1f5d BootID:44148299-a856-4066-9795-b523851b2666 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c5:fa:ea:8b:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c5:fa:ea:8b:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:fa:64:41:15:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 18:44:34.615068 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.615067    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 18:44:34.615175 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.615150    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 18:44:34.618627 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.618596    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 18:44:34.618767 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.618636    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-140.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 18:44:34.619514 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.619503    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 18:44:34.619548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.619517    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 18:44:34.619548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.619525    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:44:34.620309 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.620300    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 18:44:34.621111 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.621096    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:44:34.622271 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.622260    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 18:44:34.626929 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.626918    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:44:34.627048 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627039    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 18:44:34.627080 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627069    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 18:44:34.627111 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627085    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 18:44:34.627111 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627100    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 18:44:34.627111 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627109    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 17 18:44:34.627223 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627118    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 18:44:34.627568 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627556    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:44:34.627618 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.627577    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 18:44:34.631756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.631733    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 18:44:34.634109 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.634068    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 18:44:34.634212 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.634121    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 18:44:34.635537 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635522    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635542    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635551    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635559    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635571    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635579    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635590    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635598    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 18:44:34.635608 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635609    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 18:44:34.635859 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635617    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 18:44:34.635859 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635629    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 18:44:34.635859 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.635638    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 18:44:34.636546 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.636535    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 18:44:34.636596 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.636549    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 18:44:34.639836 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.639819    2558 server.go:1268] "Started kubelet"
Jul 17 18:44:34.639925 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.639877    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 18:44:34.640009 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.639918    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 18:44:34.640112 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.640029    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 18:44:34.640735 ip-10-0-138-140 systemd[1]: Started Kubernetes Kubelet.
Jul 17 18:44:34.640981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.640956    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-140.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 18:44:34.641858 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.641830    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 18:44:34.641925 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.641834    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 18:44:34.642039 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.642021    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 18:44:34.642674 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.642616    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 18:44:34.646842 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.646811    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 18:44:34.651855 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.650752    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803e9b4a5ff  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.639791615 +0000 UTC m=+0.402221867,LastTimestamp:2026-07-17 18:44:34.639791615 +0000 UTC m=+0.402221867,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.652174 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.652154    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 18:44:34.652174 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.652168    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 18:44:34.652316 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.652301    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:34.653198 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.653036    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 18:44:34.653515 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.653483    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 18:44:34.653515 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.653507    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 18:44:34.653658 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.653623    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 18:44:34.653658 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.653646    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 18:44:34.654637 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.654615    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 18:44:34.654702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.654639    2558 factory.go:55] Registering systemd factory
Jul 17 18:44:34.654702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.654650    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 17 18:44:34.655057 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.655044    2558 factory.go:153] Registering CRI-O factory
Jul 17 18:44:34.655110 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.655058    2558 factory.go:223] Registration of the crio container factory successfully
Jul 17 18:44:34.655110 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.655077    2558 factory.go:103] Registering Raw factory
Jul 17 18:44:34.655110 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.655086    2558 manager.go:1196] Started watching for new ooms in manager
Jul 17 18:44:34.655611 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.655595    2558 manager.go:349] Starting recovery of all containers
Jul 17 18:44:34.656523 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.656496    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 17 18:44:34.656523 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.656486    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-140.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 17 18:44:34.668037 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.667876    2558 manager.go:354] Recovery completed
Jul 17 18:44:34.673065 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.673053    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.676395 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676375    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.676464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676407    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.676464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676417    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.676866 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676853    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 18:44:34.676866 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676866    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 18:44:34.676976 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676874    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.676976 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.676880    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 18:44:34.677603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.677587    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.677660 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.677617    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.677660 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.677627    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.678640 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.678555    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803ebe326d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.676393686 +0000 UTC m=+0.438823937,LastTimestamp:2026-07-17 18:44:34.676393686 +0000 UTC m=+0.438823937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.678896 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.678884    2558 policy_none.go:49] "None policy: Start"
Jul 17 18:44:34.678956 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.678902    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 18:44:34.678956 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.678913    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 18:44:34.680829 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.680819    2558 policy_none.go:47] "Start"
Jul 17 18:44:34.687093 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.687029    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803ebe36e5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.676411996 +0000 UTC m=+0.438842247,LastTimestamp:2026-07-17 18:44:34.676411996 +0000 UTC m=+0.438842247,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.691854 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.691790    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803ebe390e5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.676420837 +0000 UTC m=+0.438851088,LastTimestamp:2026-07-17 18:44:34.676420837 +0000 UTC m=+0.438851088,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.695742 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.695658    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-140.ec2.internal.18c32803ebe326d6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803ebe326d6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.676393686 +0000 UTC m=+0.438823937,LastTimestamp:2026-07-17 18:44:34.677602355 +0000 UTC m=+0.440032609,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.696177 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.696157    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ll2wj"
Jul 17 18:44:34.705236 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.705156    2558 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-140.ec2.internal.18c32803ebe36e5c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-140.ec2.internal.18c32803ebe36e5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-140.ec2.internal,UID:ip-10-0-138-140.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-140.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-140.ec2.internal,},FirstTimestamp:2026-07-17 18:44:34.676411996 +0000 UTC m=+0.438842247,LastTimestamp:2026-07-17 18:44:34.677622271 +0000 UTC m=+0.440052522,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-140.ec2.internal,}"
Jul 17 18:44:34.707519 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.707501    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ll2wj"
Jul 17 18:44:34.709900 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.709885    2558 manager.go:337] "Starting Device Plugin manager"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.709930    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.709943    2558 server.go:87] "Starting device plugin registration server"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.710152    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.710164    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.710289    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.710382    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.710390    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.710969    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 18:44:34.734297 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.711026    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:34.765511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.765476    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 18:44:34.766808 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.766788    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 18:44:34.766896 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.766828    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 18:44:34.766896 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.766856    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 18:44:34.766958 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.766895    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 18:44:34.769204 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.769182    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:44:34.811142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.811106    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.812047 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.812029    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.812155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.812066    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.812155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.812083    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.812155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.812113    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.821639 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.821588    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.821639 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.821610    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-140.ec2.internal\": node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:34.867837 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.867809    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"]
Jul 17 18:44:34.867944 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.867878    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.868818 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.868801    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:34.869275 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.869261    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.869363 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.869290    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.869363 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.869300    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.871715 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.871702    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.871833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.871819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.871886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.871860    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.872415 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872392    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.872527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872421    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.872527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872400    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.872527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872434    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.872527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872457    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.872527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.872473    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.874727 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.874710    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.874814 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.874738    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:34.875389 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.875376    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:34.875448 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.875401    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:34.875448 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.875410    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:34.903094 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.903069    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-140.ec2.internal\" not found" node="ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.907672 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.907655    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-140.ec2.internal\" not found" node="ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.955963 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.955933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.956090 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.955966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.956090 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:34.955992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/623d991e054ce9bc2a7b170b88241ceb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"623d991e054ce9bc2a7b170b88241ceb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:34.969144 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:34.969124    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.056402 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.056526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/623d991e054ce9bc2a7b170b88241ceb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"623d991e054ce9bc2a7b170b88241ceb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.056526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.056526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.056526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c3ccb42afd9fc8cf1a1354bfa4d2fc8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal\" (UID: \"3c3ccb42afd9fc8cf1a1354bfa4d2fc8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.056526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.056517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/623d991e054ce9bc2a7b170b88241ceb-config\") pod \"kube-apiserver-proxy-ip-10-0-138-140.ec2.internal\" (UID: \"623d991e054ce9bc2a7b170b88241ceb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.069502 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.069471    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.170420 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.170349    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.206579 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.206540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.210744 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.210724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.271303 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.271271    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.371759 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.371730    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.472375 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.472299    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.572787 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.572756    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.587946 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.587925    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 18:44:35.588074 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.588060    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 18:44:35.672965 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:35.672939    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-140.ec2.internal\" not found"
Jul 17 18:44:35.714251 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.714208    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:39:35 +0000 UTC" deadline="2028-03-05 16:37:07.006073494 +0000 UTC"
Jul 17 18:44:35.714251 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.714240    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14325h52m31.291836347s"
Jul 17 18:44:35.716382 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.716364    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 18:44:35.717656 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.717633    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 18:44:35.717768 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.717665    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 18:44:35.717768 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.717679    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeHasSufficientPID"
Jul 17 18:44:35.754380 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.754286    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:44:35.803238 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:35.803088    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3ccb42afd9fc8cf1a1354bfa4d2fc8.slice/crio-855ba927ad6e3bc8a3a72a8dc50cc29425ff560d12bf003161a557e5ae761b48 WatchSource:0}: Error finding container 855ba927ad6e3bc8a3a72a8dc50cc29425ff560d12bf003161a557e5ae761b48: Status 404 returned error can't find the container with id 855ba927ad6e3bc8a3a72a8dc50cc29425ff560d12bf003161a557e5ae761b48
Jul 17 18:44:35.804109 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:35.804071    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623d991e054ce9bc2a7b170b88241ceb.slice/crio-326dc542e47992f9cb563288045caca52547d10b6bb5c5d1f7d0beb742dac258 WatchSource:0}: Error finding container 326dc542e47992f9cb563288045caca52547d10b6bb5c5d1f7d0beb742dac258: Status 404 returned error can't find the container with id 326dc542e47992f9cb563288045caca52547d10b6bb5c5d1f7d0beb742dac258
Jul 17 18:44:35.808204 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.808177    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:44:35.853116 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.853074    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.869875 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.869848    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:44:35.871610 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.871594    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal"
Jul 17 18:44:35.881285 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.881269    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 18:44:35.914987 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:35.914968    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:44:36.105345 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.105252    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:44:36.511690 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.511427    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 18:44:36.628481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.628450    2558 apiserver.go:52] "Watching apiserver"
Jul 17 18:44:36.636862 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.636837    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 18:44:36.637273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.637247    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-htt78","openshift-multus/multus-additional-cni-plugins-zztfp","openshift-multus/multus-rp7fc","openshift-multus/network-metrics-daemon-hb6vp","kube-system/global-pull-secret-syncer-wv5n9","openshift-dns/node-resolver-xcbkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal","openshift-network-diagnostics/network-check-target-sqfrw","openshift-network-operator/iptables-alerter-57rtn","openshift-ovn-kubernetes/ovnkube-node-bd5jh","kube-system/konnectivity-agent-vtlz6","kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz","openshift-cluster-node-tuning-operator/tuned-xjn72"]
Jul 17 18:44:36.640786 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.640751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.643146 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.643120    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.644096 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644074    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.644197 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644098    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 18:44:36.644197 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644118    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 18:44:36.644667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644361    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 18:44:36.644667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644396    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.644667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644387    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 18:44:36.644667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.644466    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhvpt\""
Jul 17 18:44:36.645760 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.645740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.645854 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.645825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 18:44:36.645912 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.645892    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tm5vz\""
Jul 17 18:44:36.648407 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.648384    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.648529 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.648507    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:36.649015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.648992    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.649115 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649002    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.649178 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649167    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.649238 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649206    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 18:44:36.649238 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649217    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rzt9w\""
Jul 17 18:44:36.649332 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 18:44:36.649332 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.649258    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.651022 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.651000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.651097 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.651061    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:36.653915 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.653895    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 18:44:36.654586 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.654567    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.658548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.657151    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.658548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.657711    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.658548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.657815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hh542\""
Jul 17 18:44:36.658548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.657819    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.660156 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.660135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.661481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.660933    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.661481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.661125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hm8g7\""
Jul 17 18:44:36.661481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.661284    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.661481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.661337    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 18:44:36.663213 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.663180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.664559 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.664536    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.664719 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.664699    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.665381 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665356    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vrj5m\""
Jul 17 18:44:36.665641 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-daemon-config\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.665734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-kubelet-config\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.665734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665707    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-kubelet\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.665839 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-etc-kubernetes\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.665839 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.665935 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.666002 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-socket-dir-parent\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.666002 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.665989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-script-lib\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666025    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-dbus\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.666163 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-kubelet\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666225 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv2g4\" (UniqueName: \"kubernetes.io/projected/1e5edb1e-f45b-451e-b6d6-db1114faf672-kube-api-access-wv2g4\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-env-overrides\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-os-release\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.666464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-netd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-hostroot\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.666618 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666508    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.666835 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-ovn\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.666919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-bin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.666919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ba022f-d700-4dea-83fe-3f00bd2bd1db-hosts-file\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.667033 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.666932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-log-socket\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.667033 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-system-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.667175 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqj6z\" (UniqueName: \"kubernetes.io/projected/e5660217-6572-40a9-92e2-221d340b4fd9-kube-api-access-cqj6z\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.667247 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-systemd-units\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.667304 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667282    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.667382 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.667382 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-cni-binary-copy\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.667541 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-k8s-cni-cncf-io\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.667541 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-multus-certs\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.667724 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-bin\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.667928 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovn-node-metrics-cert\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.667985 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-registration-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.668067 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.667989    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-device-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.668067 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668003    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 18:44:36.668177 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5ds\" (UniqueName: \"kubernetes.io/projected/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kube-api-access-7v5ds\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.668177 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668168    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zpd\" (UniqueName: \"kubernetes.io/projected/becdbe05-e6c9-4d52-9441-006828d07ca8-kube-api-access-z8zpd\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.668304 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668206    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897hn\" (UniqueName: \"kubernetes.io/projected/24ba022f-d700-4dea-83fe-3f00bd2bd1db-kube-api-access-897hn\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.668304 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.668304 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-sys-fs\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.668503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-cnibin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.668503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-netns\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.668503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.668503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-systemd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.668503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-var-lib-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-node-log\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-multus\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vd2\" (UniqueName: \"kubernetes.io/projected/ef151684-97e8-459b-aec6-07672b896efc-kube-api-access-v4vd2\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba022f-d700-4dea-83fe-3f00bd2bd1db-tmp-dir\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.668761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5660217-6572-40a9-92e2-221d340b4fd9-iptables-alerter-script\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668778    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5660217-6572-40a9-92e2-221d340b4fd9-host-slash\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668828    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-etc-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-slash\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-netns\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668950    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-config\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.668993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-socket-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.669125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.669076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-conf-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.669593 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.669275    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-plzgx\""
Jul 17 18:44:36.669593 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.669524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.670952 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.670932    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 18:44:36.671100 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.671069    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 18:44:36.672237 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.671817    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dwgm4\""
Jul 17 18:44:36.672237 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.671818    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 18:44:36.672570 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.672547    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-425pv\""
Jul 17 18:44:36.672932 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.672913    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 18:44:36.673246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.673227    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 18:44:36.674833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.674812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:36.674928 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.674880    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:36.704251 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.703885    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z28hg"
Jul 17 18:44:36.713325 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.713305    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z28hg"
Jul 17 18:44:36.755098 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.755066    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 18:44:36.769586 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769501    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-bin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.769586 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ba022f-d700-4dea-83fe-3f00bd2bd1db-hosts-file\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.769586 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-log-socket\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-systemd\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-system-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqj6z\" (UniqueName: \"kubernetes.io/projected/e5660217-6572-40a9-92e2-221d340b4fd9-kube-api-access-cqj6z\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-systemd-units\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-sys\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.769809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-cni-binary-copy\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24ba022f-d700-4dea-83fe-3f00bd2bd1db-hosts-file\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-k8s-cni-cncf-io\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.769992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-host\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-multus-certs\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-bin\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.770152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovn-node-metrics-cert\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-registration-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-device-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5ds\" (UniqueName: \"kubernetes.io/projected/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kube-api-access-7v5ds\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-systemd-units\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zpd\" (UniqueName: \"kubernetes.io/projected/becdbe05-e6c9-4d52-9441-006828d07ca8-kube-api-access-z8zpd\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897hn\" (UniqueName: \"kubernetes.io/projected/24ba022f-d700-4dea-83fe-3f00bd2bd1db-kube-api-access-897hn\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc37e0f-36bd-4e70-b105-fc17931f9699-konnectivity-ca\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-sys-fs\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-cnibin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-netns\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.770590 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-cni-binary-copy\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770615    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-bin\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770930    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.770962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-systemd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-tmp\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-device-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771058    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-system-cni-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-registration-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-var-lib-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-node-log\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-netns\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc37e0f-36bd-4e70-b105-fc17931f9699-agent-certs\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-node-log\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.771235 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-sys-fs\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-var-lib-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-cnibin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.771386    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.771487    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:44:37.271465852 +0000 UTC m=+3.033896114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-multus\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vd2\" (UniqueName: \"kubernetes.io/projected/ef151684-97e8-459b-aec6-07672b896efc-kube-api-access-v4vd2\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba022f-d700-4dea-83fe-3f00bd2bd1db-tmp-dir\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5660217-6572-40a9-92e2-221d340b4fd9-iptables-alerter-script\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5660217-6572-40a9-92e2-221d340b4fd9-host-slash\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-etc-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771726    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-tuned\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-slash\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-systemd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-multus-certs\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-log-socket\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771945    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-multus\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-run-k8s-cni-cncf-io\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.771994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-cni-bin\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-slash\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-system-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba022f-d700-4dea-83fe-3f00bd2bd1db-tmp-dir\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-netns\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-config\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772667    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-modprobe-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-run\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-lib-modules\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772794    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-os-release\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-socket-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772855    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-conf-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-daemon-config\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.772955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-kubelet-config\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.772993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-host\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773018    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5660217-6572-40a9-92e2-221d340b4fd9-iptables-alerter-script\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq2sz\" (UniqueName: \"kubernetes.io/projected/1c691c89-234c-4620-88b8-9bbe7d1fcf43-kube-api-access-zq2sz\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773075    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-kubelet-config\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkm6\" (UniqueName: \"kubernetes.io/projected/0572a173-ab6b-43d2-90ab-522059991d5c-kube-api-access-5dkm6\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-kubelet\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-netns\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-etc-kubernetes\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773201    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-var-lib-kubelet\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-socket-dir-parent\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-script-lib\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-serviceca\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-dbus\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-kubelet\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.773857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv2g4\" (UniqueName: \"kubernetes.io/projected/1e5edb1e-f45b-451e-b6d6-db1114faf672-kube-api-access-wv2g4\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysconfig\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-cnibin\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-env-overrides\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773663    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-config\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-socket-dir\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.773954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-conf-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovn-node-metrics-cert\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-cni-dir\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-daemon-config\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kfk\" (UniqueName: \"kubernetes.io/projected/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-kube-api-access-l6kfk\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-os-release\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-netd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.774756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-kubernetes\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-hostroot\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-ovnkube-script-lib\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-ovn\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-kubelet\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-conf\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774759    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-multus-socket-dir-parent\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-os-release\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1e5edb1e-f45b-451e-b6d6-db1114faf672-env-overrides\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5660217-6572-40a9-92e2-221d340b4fd9-host-slash\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774672    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3761999b-61c5-40c7-8d95-5c228ce395b0-dbus\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-host-var-lib-kubelet\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774820    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-hostroot\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-run-ovn\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.774868    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774874    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/becdbe05-e6c9-4d52-9441-006828d07ca8-etc-kubernetes\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.775535 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.776434 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-etc-openvswitch\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.776434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.774905    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:37.274892522 +0000 UTC m=+3.037322762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:36.776434 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.774921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1e5edb1e-f45b-451e-b6d6-db1114faf672-host-cni-netd\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.776858 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.776806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"3c3ccb42afd9fc8cf1a1354bfa4d2fc8","Type":"ContainerStarted","Data":"855ba927ad6e3bc8a3a72a8dc50cc29425ff560d12bf003161a557e5ae761b48"}
Jul 17 18:44:36.778432 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.778397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" event={"ID":"623d991e054ce9bc2a7b170b88241ceb","Type":"ContainerStarted","Data":"326dc542e47992f9cb563288045caca52547d10b6bb5c5d1f7d0beb742dac258"}
Jul 17 18:44:36.783150 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.783130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqj6z\" (UniqueName: \"kubernetes.io/projected/e5660217-6572-40a9-92e2-221d340b4fd9-kube-api-access-cqj6z\") pod \"iptables-alerter-57rtn\" (UID: \"e5660217-6572-40a9-92e2-221d340b4fd9\") " pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.787116 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.787037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zpd\" (UniqueName: \"kubernetes.io/projected/becdbe05-e6c9-4d52-9441-006828d07ca8-kube-api-access-z8zpd\") pod \"multus-rp7fc\" (UID: \"becdbe05-e6c9-4d52-9441-006828d07ca8\") " pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.787116 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.787044    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv2g4\" (UniqueName: \"kubernetes.io/projected/1e5edb1e-f45b-451e-b6d6-db1114faf672-kube-api-access-wv2g4\") pod \"ovnkube-node-bd5jh\" (UID: \"1e5edb1e-f45b-451e-b6d6-db1114faf672\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.788426 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.788378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-897hn\" (UniqueName: \"kubernetes.io/projected/24ba022f-d700-4dea-83fe-3f00bd2bd1db-kube-api-access-897hn\") pod \"node-resolver-xcbkz\" (UID: \"24ba022f-d700-4dea-83fe-3f00bd2bd1db\") " pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.789941 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.789890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vd2\" (UniqueName: \"kubernetes.io/projected/ef151684-97e8-459b-aec6-07672b896efc-kube-api-access-v4vd2\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:36.791600 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.791573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5ds\" (UniqueName: \"kubernetes.io/projected/a86a6f6d-dd7e-4d50-94dc-45c74212cb1c-kube-api-access-7v5ds\") pod \"aws-ebs-csi-driver-node-8hrwz\" (UID: \"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.875015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.874982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-tmp\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.875015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-system-cni-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc37e0f-36bd-4e70-b105-fc17931f9699-agent-certs\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-tuned\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-modprobe-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.875260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-system-cni-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.875759 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-modprobe-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.875894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875772    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.875961 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-run\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876011 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.875994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-lib-modules\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-os-release\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.876230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-run\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876296 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-lib-modules\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876296 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-host\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876428 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876319    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-host\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876428 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-os-release\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.876540 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876407    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq2sz\" (UniqueName: \"kubernetes.io/projected/1c691c89-234c-4620-88b8-9bbe7d1fcf43-kube-api-access-zq2sz\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876599 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkm6\" (UniqueName: \"kubernetes.io/projected/0572a173-ab6b-43d2-90ab-522059991d5c-kube-api-access-5dkm6\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.876599 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-var-lib-kubelet\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876715 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-serviceca\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.876715 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876658    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysconfig\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876715 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.876930 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876759    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-cnibin\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.876930 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0572a173-ab6b-43d2-90ab-522059991d5c-cnibin\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.877030 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.876943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysconfig\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877130 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-d\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877258 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-serviceca\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.877323 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.877378 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.877464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877377    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kfk\" (UniqueName: \"kubernetes.io/projected/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-kube-api-access-l6kfk\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.877464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-kubernetes\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877449    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-var-lib-kubelet\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-conf\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-systemd\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-sys\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.877783 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877587    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.877783 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-host\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.877783 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.877662    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc37e0f-36bd-4e70-b105-fc17931f9699-konnectivity-ca\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.878045 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.878154 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-systemd\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.878220 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-kubernetes\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.878275 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2dc37e0f-36bd-4e70-b105-fc17931f9699-konnectivity-ca\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.878356 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-sys\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.878458 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-sysctl-conf\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.878458 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.878572 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-host\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.878828 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2dc37e0f-36bd-4e70-b105-fc17931f9699-agent-certs\") pod \"konnectivity-agent-vtlz6\" (UID: \"2dc37e0f-36bd-4e70-b105-fc17931f9699\") " pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:36.878901 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.878872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0572a173-ab6b-43d2-90ab-522059991d5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.880539 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.880515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-tmp\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.881238 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.881219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1c691c89-234c-4620-88b8-9bbe7d1fcf43-etc-tuned\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.884683 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.884663    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:36.884732 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.884691    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:36.884732 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.884707    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:36.884800 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:36.884777    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:37.384757481 +0000 UTC m=+3.147187751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:36.886955 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.886926    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq2sz\" (UniqueName: \"kubernetes.io/projected/1c691c89-234c-4620-88b8-9bbe7d1fcf43-kube-api-access-zq2sz\") pod \"tuned-xjn72\" (UID: \"1c691c89-234c-4620-88b8-9bbe7d1fcf43\") " pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:36.887545 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.887526    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkm6\" (UniqueName: \"kubernetes.io/projected/0572a173-ab6b-43d2-90ab-522059991d5c-kube-api-access-5dkm6\") pod \"multus-additional-cni-plugins-zztfp\" (UID: \"0572a173-ab6b-43d2-90ab-522059991d5c\") " pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:36.887835 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.887818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kfk\" (UniqueName: \"kubernetes.io/projected/f4c47689-ae80-4e61-bb80-f4cb8f3dfadf-kube-api-access-l6kfk\") pod \"node-ca-htt78\" (UID: \"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf\") " pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:36.959082 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.959043    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:44:36.969145 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.969096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz"
Jul 17 18:44:36.972859 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:36.972824    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5edb1e_f45b_451e_b6d6_db1114faf672.slice/crio-43b73127ae8b9547eec23801b2a6eb5c5d972d08489dfdb96730e2953d7433e2 WatchSource:0}: Error finding container 43b73127ae8b9547eec23801b2a6eb5c5d972d08489dfdb96730e2953d7433e2: Status 404 returned error can't find the container with id 43b73127ae8b9547eec23801b2a6eb5c5d972d08489dfdb96730e2953d7433e2
Jul 17 18:44:36.980810 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.980782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rp7fc"
Jul 17 18:44:36.981021 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:36.980988    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86a6f6d_dd7e_4d50_94dc_45c74212cb1c.slice/crio-8113cdaba756f52662520eadb46311cc575272810dc5bd47202025a7d3583059 WatchSource:0}: Error finding container 8113cdaba756f52662520eadb46311cc575272810dc5bd47202025a7d3583059: Status 404 returned error can't find the container with id 8113cdaba756f52662520eadb46311cc575272810dc5bd47202025a7d3583059
Jul 17 18:44:36.986037 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.986007    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcbkz"
Jul 17 18:44:36.995525 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:36.995505    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-57rtn"
Jul 17 18:44:36.997291 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:36.997262    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecdbe05_e6c9_4d52_9441_006828d07ca8.slice/crio-1f9c4b45b284018d0c1d906fcae290491ed93ac033204998f83a45fa2cb33644 WatchSource:0}: Error finding container 1f9c4b45b284018d0c1d906fcae290491ed93ac033204998f83a45fa2cb33644: Status 404 returned error can't find the container with id 1f9c4b45b284018d0c1d906fcae290491ed93ac033204998f83a45fa2cb33644
Jul 17 18:44:37.001001 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:37.000967    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ba022f_d700_4dea_83fe_3f00bd2bd1db.slice/crio-c2af769758764c58c8fa058aae7f9daaef385c79aa7e06c9cfdb459ba535abd5 WatchSource:0}: Error finding container c2af769758764c58c8fa058aae7f9daaef385c79aa7e06c9cfdb459ba535abd5: Status 404 returned error can't find the container with id c2af769758764c58c8fa058aae7f9daaef385c79aa7e06c9cfdb459ba535abd5
Jul 17 18:44:37.004872 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.004846    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xjn72"
Jul 17 18:44:37.006725 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:37.006703    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5660217_6572_40a9_92e2_221d340b4fd9.slice/crio-7ed67fcea58750f03f6e8d928fc1a512d632482a0c8aafdd052332e720a74404 WatchSource:0}: Error finding container 7ed67fcea58750f03f6e8d928fc1a512d632482a0c8aafdd052332e720a74404: Status 404 returned error can't find the container with id 7ed67fcea58750f03f6e8d928fc1a512d632482a0c8aafdd052332e720a74404
Jul 17 18:44:37.011458 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.011433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zztfp"
Jul 17 18:44:37.020183 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.020118    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-htt78"
Jul 17 18:44:37.027356 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.027313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:37.282082 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.281998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:37.282082 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.282049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:37.282292 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.282164    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:37.282292 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.282188    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:37.282292 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.282237    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:38.282220828 +0000 UTC m=+4.044651067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:37.282292 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.282255    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:44:38.282247259 +0000 UTC m=+4.044677498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:37.483692 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.483650    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:37.483882 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.483834    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:37.483882 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.483859    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:37.483882 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.483873    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:37.484041 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:37.483937    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:38.483918661 +0000 UTC m=+4.246348925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:37.717536 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.717440    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:39:37 +0000 UTC" deadline="2028-03-02 08:58:13.49957409 +0000 UTC"
Jul 17 18:44:37.717536 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.717482    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14246h13m35.782096085s"
Jul 17 18:44:37.781192 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.781155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57rtn" event={"ID":"e5660217-6572-40a9-92e2-221d340b4fd9","Type":"ContainerStarted","Data":"7ed67fcea58750f03f6e8d928fc1a512d632482a0c8aafdd052332e720a74404"}
Jul 17 18:44:37.782448 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.782415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcbkz" event={"ID":"24ba022f-d700-4dea-83fe-3f00bd2bd1db","Type":"ContainerStarted","Data":"c2af769758764c58c8fa058aae7f9daaef385c79aa7e06c9cfdb459ba535abd5"}
Jul 17 18:44:37.783647 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.783622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rp7fc" event={"ID":"becdbe05-e6c9-4d52-9441-006828d07ca8","Type":"ContainerStarted","Data":"1f9c4b45b284018d0c1d906fcae290491ed93ac033204998f83a45fa2cb33644"}
Jul 17 18:44:37.784861 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.784834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz" event={"ID":"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c","Type":"ContainerStarted","Data":"8113cdaba756f52662520eadb46311cc575272810dc5bd47202025a7d3583059"}
Jul 17 18:44:37.786089 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:37.786064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"43b73127ae8b9547eec23801b2a6eb5c5d972d08489dfdb96730e2953d7433e2"}
Jul 17 18:44:38.155261 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:38.155200    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c691c89_234c_4620_88b8_9bbe7d1fcf43.slice/crio-38468c651af7242d892032ec844f05f484b456caec1ac03b3c7d776bcf9452cb WatchSource:0}: Error finding container 38468c651af7242d892032ec844f05f484b456caec1ac03b3c7d776bcf9452cb: Status 404 returned error can't find the container with id 38468c651af7242d892032ec844f05f484b456caec1ac03b3c7d776bcf9452cb
Jul 17 18:44:38.168560 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:38.168535    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c47689_ae80_4e61_bb80_f4cb8f3dfadf.slice/crio-c9f659c0b3ed1057a25eb63c1af81528526ca76ec5fdf26782197638871caf60 WatchSource:0}: Error finding container c9f659c0b3ed1057a25eb63c1af81528526ca76ec5fdf26782197638871caf60: Status 404 returned error can't find the container with id c9f659c0b3ed1057a25eb63c1af81528526ca76ec5fdf26782197638871caf60
Jul 17 18:44:38.169782 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:38.169752    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc37e0f_36bd_4e70_b105_fc17931f9699.slice/crio-573cd60a884a1a9910d9287dce9ab76702d0dd93bf150fcfd293d67f0c94590d WatchSource:0}: Error finding container 573cd60a884a1a9910d9287dce9ab76702d0dd93bf150fcfd293d67f0c94590d: Status 404 returned error can't find the container with id 573cd60a884a1a9910d9287dce9ab76702d0dd93bf150fcfd293d67f0c94590d
Jul 17 18:44:38.171693 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:44:38.171352    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0572a173_ab6b_43d2_90ab_522059991d5c.slice/crio-4010a3148e79e9242db7f7913c0c8edcd29afc49b49cdf1dbe8b77f0bedeb97b WatchSource:0}: Error finding container 4010a3148e79e9242db7f7913c0c8edcd29afc49b49cdf1dbe8b77f0bedeb97b: Status 404 returned error can't find the container with id 4010a3148e79e9242db7f7913c0c8edcd29afc49b49cdf1dbe8b77f0bedeb97b
Jul 17 18:44:38.289406 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.289184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:38.289588 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.289479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:38.289588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.289385    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:38.289588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.289539    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:38.289588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.289574    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:44:40.289554185 +0000 UTC m=+6.051984425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:38.289811 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.289594    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:40.289580729 +0000 UTC m=+6.052010973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:38.491825 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.491791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:38.492001 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.491923    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:38.492001 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.491940    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:38.492001 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.491949    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:38.492001 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.492003    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:40.491988893 +0000 UTC m=+6.254419133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:38.717843 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.717746    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:39:37 +0000 UTC" deadline="2028-04-21 12:08:26.789569796 +0000 UTC"
Jul 17 18:44:38.717843 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.717791    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15449h23m48.071783308s"
Jul 17 18:44:38.773625 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.773588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:38.773834 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.773784    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:38.773942 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.773915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:38.774121 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.774095    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:38.774186 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.773603    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:38.774256 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:38.774236    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:38.790992 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.790946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtlz6" event={"ID":"2dc37e0f-36bd-4e70-b105-fc17931f9699","Type":"ContainerStarted","Data":"573cd60a884a1a9910d9287dce9ab76702d0dd93bf150fcfd293d67f0c94590d"}
Jul 17 18:44:38.799510 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.799480    2558 generic.go:356] "Generic (PLEG): container finished" podID="3c3ccb42afd9fc8cf1a1354bfa4d2fc8" containerID="8fa3ea50d9c84bdfcd9c1c7b54844c6d760a6a94648e6a04f5f5d73a0cd25248" exitCode=0
Jul 17 18:44:38.799839 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.799751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"3c3ccb42afd9fc8cf1a1354bfa4d2fc8","Type":"ContainerDied","Data":"8fa3ea50d9c84bdfcd9c1c7b54844c6d760a6a94648e6a04f5f5d73a0cd25248"}
Jul 17 18:44:38.817830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.817797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htt78" event={"ID":"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf","Type":"ContainerStarted","Data":"c9f659c0b3ed1057a25eb63c1af81528526ca76ec5fdf26782197638871caf60"}
Jul 17 18:44:38.835060 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.834973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjn72" event={"ID":"1c691c89-234c-4620-88b8-9bbe7d1fcf43","Type":"ContainerStarted","Data":"38468c651af7242d892032ec844f05f484b456caec1ac03b3c7d776bcf9452cb"}
Jul 17 18:44:38.843691 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.843632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" event={"ID":"623d991e054ce9bc2a7b170b88241ceb","Type":"ContainerStarted","Data":"0c73f3b19461c7160389cf477566bd9f6df126c6328f8e6e03bb123e585e7d1b"}
Jul 17 18:44:38.846388 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:38.846360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerStarted","Data":"4010a3148e79e9242db7f7913c0c8edcd29afc49b49cdf1dbe8b77f0bedeb97b"}
Jul 17 18:44:39.873789 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:39.873750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" event={"ID":"3c3ccb42afd9fc8cf1a1354bfa4d2fc8","Type":"ContainerStarted","Data":"6cbb4030deba4a8c47a6518c54a5f66aab7898b16e637bd4ab18cad1f80e3578"}
Jul 17 18:44:39.897149 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:39.895948    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-140.ec2.internal" podStartSLOduration=3.895924337 podStartE2EDuration="3.895924337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:38.864322351 +0000 UTC m=+4.626752638" watchObservedRunningTime="2026-07-17 18:44:39.895924337 +0000 UTC m=+5.658354601"
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.307651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.307727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.307890    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.307956    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:44:44.307936926 +0000 UTC m=+10.070367170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.308373    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:40.308541 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.308450    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:44.308433534 +0000 UTC m=+10.070863777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:40.510636 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.509948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:40.510636 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.510142    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:40.510636 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.510169    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:40.510636 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.510182    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:40.510636 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.510247    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:44.510228899 +0000 UTC m=+10.272659144 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:40.768438 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.767808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:40.768438 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.767936    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:40.768438 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.768318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:40.768438 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:40.768318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:40.768772 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.768462    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:40.768772 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:40.768551    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:42.767514 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:42.767476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:42.767978 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:42.767474    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:42.767978 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:42.767621    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:42.767978 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:42.767644    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:42.767978 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:42.767789    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:42.767978 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:42.767894    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:44.344678 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.344634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:44.345168 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.344705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:44.345168 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.344857    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:44.345168 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.344920    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:44:52.344899848 +0000 UTC m=+18.107330107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:44.345337 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.345311    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:44.345404 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.345355    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:52.345341056 +0000 UTC m=+18.107771310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:44.546080 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.546041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:44.546245 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.546184    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:44.546245 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.546199    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:44.546245 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.546208    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:44.546427 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.546262    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:44:52.54624406 +0000 UTC m=+18.308674301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:44.770426 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.770299    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:44.770426 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.770419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:44.770631 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.770429    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:44.770631 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:44.770456    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:44.770631 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.770556    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:44.770784 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:44.770658    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:46.771301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:46.771272    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:46.771785 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:46.771273    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:46.771785 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:46.771398    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:46.771785 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:46.771513    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:46.771785 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:46.771276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:46.771785 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:46.771607    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:48.770672 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:48.770643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:48.771122 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:48.770746    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:48.771122 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:48.770753    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:48.771122 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:48.770837    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:48.771122 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:48.770885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:48.771122 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:48.770969    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:49.917468 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:49.917433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz" event={"ID":"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c","Type":"ContainerStarted","Data":"bcb67731f9f31f863787320a7465da0da9517ea29d5c7e946d9873d71ade2d48"}
Jul 17 18:44:50.767902 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:50.767871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:50.768108 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:50.767872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:50.768108 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:50.767992    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:50.768108 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:50.768005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:50.768281 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:50.768110    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:50.768281 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:50.768222    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:52.407718 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.407679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:52.408192 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.407738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:52.408192 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.407855    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:52.408192 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.407863    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:52.408192 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.407920    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:45:08.407901595 +0000 UTC m=+34.170331833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:44:52.408192 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.407939    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:08.40793209 +0000 UTC m=+34.170362328 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:44:52.609229 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.609191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:52.609431 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.609378    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:44:52.609431 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.609399    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:44:52.609431 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.609412    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:52.609589 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.609469    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:08.609451076 +0000 UTC m=+34.371881332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:44:52.767680 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.767582    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:52.767846 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.767582    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:52.767846 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.767743    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:52.767846 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:52.767596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:52.767846 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.767796    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:52.768036 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:52.767859    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:54.768484 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:54.768402    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:54.768924 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:54.768519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:54.768924 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:54.768613    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:54.768924 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:54.768515    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:54.768924 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:54.768652    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:54.768924 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:54.768754    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:56.767254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:56.767217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:56.767752 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:56.767374    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:56.767752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:56.767217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:56.767752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:56.767217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:56.767752 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:56.767477    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:56.767752 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:56.767554    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:57.895985 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.895566    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 18:44:57.933587 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.933553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz" event={"ID":"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c","Type":"ContainerStarted","Data":"448ef9c6ec5e155bde59246d63e6cdc751ebb899f6ad1d452dfe4ca2a3bb1f92"}
Jul 17 18:44:57.935239 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.935212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"3e30f2f346cf5e0621e755e16f24c5fd4c98c309b87e8d98676b02e5dab4a33a"}
Jul 17 18:44:57.935378 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.935276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"63b6da0d7b7fda6fd561effe1daf562f89342a6004628e079c0f592557ac87bb"}
Jul 17 18:44:57.937460 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.937427    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="5a86a0117c9d7eb2aec4d851c7205274dac1edf4252ce891a1d8e3a284bdce9d" exitCode=0
Jul 17 18:44:57.937572 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.937552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"5a86a0117c9d7eb2aec4d851c7205274dac1edf4252ce891a1d8e3a284bdce9d"}
Jul 17 18:44:57.939131 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.939071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vtlz6" event={"ID":"2dc37e0f-36bd-4e70-b105-fc17931f9699","Type":"ContainerStarted","Data":"fe99207d2784267b5b693304c10218d0595c69a1b2b31c410c3f6df8fc9ed612"}
Jul 17 18:44:57.940752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.940731    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcbkz" event={"ID":"24ba022f-d700-4dea-83fe-3f00bd2bd1db","Type":"ContainerStarted","Data":"95d65a3b2c53dcdb413ee0ca3b4941f64932538acb10b27fd68f5b5a17704cc3"}
Jul 17 18:44:57.943412 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.943389    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rp7fc" event={"ID":"becdbe05-e6c9-4d52-9441-006828d07ca8","Type":"ContainerStarted","Data":"65013664e0a50ec9236ab3e90e4fd8e0524fcd931634bed3d9b60cdad77c1edd"}
Jul 17 18:44:57.945312 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.945284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-htt78" event={"ID":"f4c47689-ae80-4e61-bb80-f4cb8f3dfadf","Type":"ContainerStarted","Data":"2682b3478f31bf10da4fa86b2aab4c748a3a98ee3c8a1c1f05a3d27540a7ab15"}
Jul 17 18:44:57.946895 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.946871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xjn72" event={"ID":"1c691c89-234c-4620-88b8-9bbe7d1fcf43","Type":"ContainerStarted","Data":"ddc7d3854df74f8a3df12d5e97d804c2214007bd10e8542817c398678e374567"}
Jul 17 18:44:57.961361 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.961292    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-140.ec2.internal" podStartSLOduration=21.961276996 podStartE2EDuration="21.961276996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:39.897743753 +0000 UTC m=+5.660174046" watchObservedRunningTime="2026-07-17 18:44:57.961276996 +0000 UTC m=+23.723707247"
Jul 17 18:44:57.981479 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:57.981422    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rp7fc" podStartSLOduration=2.324073607 podStartE2EDuration="22.981408585s" totalImagesPullingTime="20.657334978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:57.980929358 +0000 UTC m=+23.743359619" watchObservedRunningTime="2026-07-17 18:44:57.981408585 +0000 UTC m=+23.743838844"
Jul 17 18:44:58.002773 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.002724    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xjn72" podStartSLOduration=3.50647681 podStartE2EDuration="23.002709108s" totalImagesPullingTime="19.496232298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:58.002422813 +0000 UTC m=+23.764853073" watchObservedRunningTime="2026-07-17 18:44:58.002709108 +0000 UTC m=+23.765139368"
Jul 17 18:44:58.019848 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.019803    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-htt78" podStartSLOduration=12.067291038 podStartE2EDuration="23.019784669s" totalImagesPullingTime="10.952493631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:58.019724805 +0000 UTC m=+23.782155066" watchObservedRunningTime="2026-07-17 18:44:58.019784669 +0000 UTC m=+23.782214931"
Jul 17 18:44:58.044826 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.044774    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vtlz6" podStartSLOduration=12.094092875 podStartE2EDuration="23.044757487s" totalImagesPullingTime="10.950664612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:58.043917418 +0000 UTC m=+23.806347680" watchObservedRunningTime="2026-07-17 18:44:58.044757487 +0000 UTC m=+23.807187747"
Jul 17 18:44:58.075876 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.075829    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xcbkz" podStartSLOduration=10.956354167 podStartE2EDuration="23.075814012s" totalImagesPullingTime="12.119459845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:58.075365941 +0000 UTC m=+23.837796202" watchObservedRunningTime="2026-07-17 18:44:58.075814012 +0000 UTC m=+23.838244270"
Jul 17 18:44:58.724553 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.724426    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:44:57.89596112Z","UUID":"41326d1d-9e00-4367-a791-0ee41c56e186","Handler":null,"Name":"","Endpoint":""}
Jul 17 18:44:58.726039 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.726013    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 18:44:58.726167 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.726056    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 18:44:58.768013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.767986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:44:58.768013 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.767999    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:44:58.768258 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.768108    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:44:58.768258 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:58.768120    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:44:58.768258 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:58.768199    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:44:58.768445 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:44:58.768270    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:44:58.805050 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.805019    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:58.805757 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.805736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:58.954634 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.954555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"812e5334b3f03e21d0b78b1473869e1c4ba45b13feca2630c0bfed859b22cc21"}
Jul 17 18:44:58.954634 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.954625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"aa44ef2f432eff84357a18d094df15738e1a88282b628b91470b7e116db9a20c"}
Jul 17 18:44:58.955177 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.954656    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"550307af8f4a98fbef411b5d493629fc0244a7f63a5845bc46f511d9b19d5e03"}
Jul 17 18:44:58.955177 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:58.954686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"f975cad1d3a6b2b285aa8ae1ecab6e657ef4a079101f4abcf3408f605f6c0458"}
Jul 17 18:44:59.439346 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.439293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:59.440098 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.440077    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vtlz6"
Jul 17 18:44:59.958492 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.958447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-57rtn" event={"ID":"e5660217-6572-40a9-92e2-221d340b4fd9","Type":"ContainerStarted","Data":"3eceaddc0e50154e6d1e7ce525fdebb5eacc25fe0fc055c61750080bb3719c73"}
Jul 17 18:44:59.960584 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.960558    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz" event={"ID":"a86a6f6d-dd7e-4d50-94dc-45c74212cb1c","Type":"ContainerStarted","Data":"5282f4800e923d9353a4d4d612057cc6cab2a95efa9f2c5f0878f3ff0f3df752"}
Jul 17 18:44:59.975727 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.975681    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-57rtn" podStartSLOduration=12.85842352 podStartE2EDuration="24.975667457s" totalImagesPullingTime="12.117243937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:59.975280485 +0000 UTC m=+25.737710744" watchObservedRunningTime="2026-07-17 18:44:59.975667457 +0000 UTC m=+25.738097717"
Jul 17 18:44:59.998368 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:44:59.998294    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hrwz" podStartSLOduration=2.816430409 podStartE2EDuration="24.998278586s" totalImagesPullingTime="22.181848177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:59.997346957 +0000 UTC m=+25.759777237" watchObservedRunningTime="2026-07-17 18:44:59.998278586 +0000 UTC m=+25.760708845"
Jul 17 18:45:00.767668 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:00.767579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:00.767668 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:00.767657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:00.767888 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:00.767680    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:00.767888 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:00.767790    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:00.768000 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:00.767907    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:00.768000 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:00.767979    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:00.966102 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:00.966064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"29c9b4afa435b52e07077012743c50880d573755d6e68fe31762f3f8bbcb96d6"}
Jul 17 18:45:02.767470 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:02.767186    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:02.767950 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:02.767265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:02.767950 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:02.767580    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:02.767950 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:02.767301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:02.767950 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:02.767634    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:02.767950 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:02.767729    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:03.977534 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:03.977262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" event={"ID":"1e5edb1e-f45b-451e-b6d6-db1114faf672","Type":"ContainerStarted","Data":"37221240bc74b2e0e5afde788f880508b413f141bba5aeeceb98a29d7efd9d0c"}
Jul 17 18:45:03.978263 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:03.977640    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:03.979161 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:03.979134    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="ee2411ee34bc6348e2010d3fac52487ceac114b6ff25b7047af4be29e8630995" exitCode=0
Jul 17 18:45:03.979275 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:03.979182    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"ee2411ee34bc6348e2010d3fac52487ceac114b6ff25b7047af4be29e8630995"}
Jul 17 18:45:03.993718 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:03.993694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:04.007594 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.007549    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh" podStartSLOduration=8.327030135 podStartE2EDuration="29.007534762s" totalImagesPullingTime="20.680504627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:04.00725894 +0000 UTC m=+29.769689201" watchObservedRunningTime="2026-07-17 18:45:04.007534762 +0000 UTC m=+29.769965021"
Jul 17 18:45:04.772540 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.771887    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:04.772540 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:04.772021    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:04.772540 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.772377    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:04.772540 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.772519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:04.772965 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:04.772538    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:04.772965 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:04.772683    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:04.981491 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.981461    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:04.981987 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.981525    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:04.996930 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:04.996903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:05.762784 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.762754    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wv5n9"]
Jul 17 18:45:05.762934 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.762860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:05.763014 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:05.762984    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:05.766064 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.766042    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hb6vp"]
Jul 17 18:45:05.766171 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.766144    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:05.766266 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:05.766244    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:05.769370 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.769351    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sqfrw"]
Jul 17 18:45:05.769459 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.769417    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:05.769505 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:05.769482    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:05.985492 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.985459    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="8496e72bfd7f440e50e682d0c234c27691845ec32ef527897c1a318ca21d88b8" exitCode=0
Jul 17 18:45:05.985950 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:05.985543    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"8496e72bfd7f440e50e682d0c234c27691845ec32ef527897c1a318ca21d88b8"}
Jul 17 18:45:07.768192 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:07.768009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:07.768615 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:07.768061    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:07.768615 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:07.768074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:07.768615 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:07.768388    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:07.768615 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:07.768264    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:07.768615 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:07.768438    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:07.992372 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:07.992271    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="bf691b441455260abc7be6360d4e8f749475f039e582bb649b5a3271e3b2d4b5" exitCode=0
Jul 17 18:45:07.992372 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:07.992345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"bf691b441455260abc7be6360d4e8f749475f039e582bb649b5a3271e3b2d4b5"}
Jul 17 18:45:08.427977 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:08.427940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:08.428189 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:08.427999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:08.428189 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.428089    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 18:45:08.428189 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.428111    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:45:08.428189 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.428143    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret podName:3761999b-61c5-40c7-8d95-5c228ce395b0 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:40.428129909 +0000 UTC m=+66.190560147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret") pod "global-pull-secret-syncer-wv5n9" (UID: "3761999b-61c5-40c7-8d95-5c228ce395b0") : object "kube-system"/"original-pull-secret" not registered
Jul 17 18:45:08.428189 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.428176    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs podName:ef151684-97e8-459b-aec6-07672b896efc nodeName:}" failed. No retries permitted until 2026-07-17 18:45:40.428157637 +0000 UTC m=+66.190587884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs") pod "network-metrics-daemon-hb6vp" (UID: "ef151684-97e8-459b-aec6-07672b896efc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 18:45:08.629607 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:08.629569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:08.629760 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.629713    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 18:45:08.629760 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.629734    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 18:45:08.629760 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.629748    2558 projected.go:196] Error preparing data for projected volume kube-api-access-d4f92 for pod openshift-network-diagnostics/network-check-target-sqfrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:45:08.629895 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:08.629800    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92 podName:d0b8270f-1579-4924-9d71-8aeb28490f59 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:40.629787204 +0000 UTC m=+66.392217442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d4f92" (UniqueName: "kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92") pod "network-check-target-sqfrw" (UID: "d0b8270f-1579-4924-9d71-8aeb28490f59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 18:45:09.768118 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:09.768034    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:09.768118 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:09.768081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:09.768701 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:09.768053    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:09.768701 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:09.768175    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hb6vp" podUID="ef151684-97e8-459b-aec6-07672b896efc"
Jul 17 18:45:09.768701 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:09.768267    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfrw" podUID="d0b8270f-1579-4924-9d71-8aeb28490f59"
Jul 17 18:45:09.768701 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:09.768347    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wv5n9" podUID="3761999b-61c5-40c7-8d95-5c228ce395b0"
Jul 17 18:45:11.600841 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.600808    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-140.ec2.internal" event="NodeReady"
Jul 17 18:45:11.601428 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.600963    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 18:45:11.646723 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.646681    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"]
Jul 17 18:45:11.650548 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.650519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"
Jul 17 18:45:11.653587 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.653566    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cm2gp"]
Jul 17 18:45:11.654949 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.654466    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.654949 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.654781    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fqszg\""
Jul 17 18:45:11.655225 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.655210    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.657504 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.657481    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"]
Jul 17 18:45:11.657889 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.657867    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.662147 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.661264    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"]
Jul 17 18:45:11.662147 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.661401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.664916 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.664897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.669143 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.669122    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.669143 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.669157    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 17 18:45:11.669340 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.669183    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 17 18:45:11.669340 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.669214    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.669340 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.669121    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.670041 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670022    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 17 18:45:11.670133 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670048    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2dn7s\""
Jul 17 18:45:11.670133 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670049    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.670237 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670138    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 17 18:45:11.670237 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670145    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.670237 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pj72r\""
Jul 17 18:45:11.670412 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670312    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-x64bz\""
Jul 17 18:45:11.670412 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670405    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 17 18:45:11.670524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670508    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 17 18:45:11.670595 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.670581    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.683867 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.683724    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cm2gp"]
Jul 17 18:45:11.684182 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.684071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"]
Jul 17 18:45:11.686365 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.686345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"]
Jul 17 18:45:11.686440 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.686392    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"]
Jul 17 18:45:11.689884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.689862    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 17 18:45:11.690583 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.690566    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5c6cb4dc5b-f5hkm"]
Jul 17 18:45:11.690719 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.690702    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.693911 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.693895    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wrcwt"]
Jul 17 18:45:11.694025 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.694009    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.695894    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696091    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696332    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696456    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696627    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 17 18:45:11.696850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696696    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.697158 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.696867    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-czwj6\""
Jul 17 18:45:11.697248 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.697217    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.697661 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.697644    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 17 18:45:11.699028 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.698968    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"]
Jul 17 18:45:11.699230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.699211    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.701517 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.701495    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 17 18:45:11.701611 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.701524    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 17 18:45:11.701778 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.701763    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kmznd\""
Jul 17 18:45:11.701852 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.701801    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hshmv\""
Jul 17 18:45:11.702047 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.702029    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 17 18:45:11.702234 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.702218    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.702306 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.702265    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.702638 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.702622    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 17 18:45:11.703444 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.703426    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:45:11.703551 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.703533    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.706653 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.706624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 17 18:45:11.706856 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.706836    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.706939 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.706872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.707010 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.706952    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f8z7z\""
Jul 17 18:45:11.709445 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.709429    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.712547 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.712525    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"]
Jul 17 18:45:11.713181 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.713160    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hph6v"]
Jul 17 18:45:11.714210 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.714191    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 17 18:45:11.715668 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.715650    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 18:45:11.715909 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.715882    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 18:45:11.716120 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.716104    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 18:45:11.719614 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.719392    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xpshk"]
Jul 17 18:45:11.719614 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.719546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.720392 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.720369    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4m4h7\""
Jul 17 18:45:11.722906 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.722882    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 18:45:11.723001 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.722882    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zt9vq\""
Jul 17 18:45:11.723063 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.723028    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"]
Jul 17 18:45:11.723162 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.723146    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.723206 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.723194    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 18:45:11.726398 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726381    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"]
Jul 17 18:45:11.726471 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726423    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c6cb4dc5b-f5hkm"]
Jul 17 18:45:11.726538 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726481    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wrcwt"]
Jul 17 18:45:11.726538 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"
Jul 17 18:45:11.726925 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726907    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.726981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.726949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 18:45:11.727733 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.727717    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2pjb8\""
Jul 17 18:45:11.727798 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.727717    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.728708 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.728691    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 18:45:11.731189 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.731166    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 18:45:11.732154 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.732136    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"]
Jul 17 18:45:11.732632 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.732604    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lgpwj\""
Jul 17 18:45:11.732815 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.732801    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 18:45:11.732965 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.732947    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hph6v"]
Jul 17 18:45:11.736827 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.736618    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"]
Jul 17 18:45:11.748081 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.748031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:45:11.753247 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bae47ed-4f68-4b6f-b482-6ed4210d1118-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.753387 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpnt\" (UniqueName: \"kubernetes.io/projected/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-kube-api-access-wmpnt\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.753387 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bae47ed-4f68-4b6f-b482-6ed4210d1118-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.753387 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6e23da7-45bb-4993-ba2b-e82954181fbb-config\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.753387 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753385    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-tmp\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.753604 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-snapshots\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.753604 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-serving-cert\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.753604 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e23da7-45bb-4993-ba2b-e82954181fbb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.753604 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p7zp\" (UniqueName: \"kubernetes.io/projected/d6e23da7-45bb-4993-ba2b-e82954181fbb-kube-api-access-2p7zp\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.753794 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jfp\" (UniqueName: \"kubernetes.io/projected/2bae47ed-4f68-4b6f-b482-6ed4210d1118-kube-api-access-c6jfp\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.753794 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753637    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbsr\" (UniqueName: \"kubernetes.io/projected/cc2303f5-dba3-4879-969c-d31129b82416-kube-api-access-dxbsr\") pod \"volume-data-source-validator-78cf4c6f6-ntj9r\" (UID: \"cc2303f5-dba3-4879-969c-d31129b82416\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"
Jul 17 18:45:11.753794 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.753794 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.753688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.765878 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.765851    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpshk"]
Jul 17 18:45:11.767095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.767076    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:11.767200 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.767096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:11.767200 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.767096    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:11.769589 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.769567    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:45:11.769785 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.769758    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jqp7v\""
Jul 17 18:45:11.769874 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.769815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktbms\""
Jul 17 18:45:11.770078 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.770059    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:45:11.854236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.854236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.854236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjfn\" (UniqueName: \"kubernetes.io/projected/c4898006-61dc-42b4-a9d6-b3022d1adc4f-kube-api-access-wdjfn\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jfp\" (UniqueName: \"kubernetes.io/projected/2bae47ed-4f68-4b6f-b482-6ed4210d1118-kube-api-access-c6jfp\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-config\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854386    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bz9\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d66827-0a83-434c-b174-8224d6fa5830-serving-cert\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.854528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bae47ed-4f68-4b6f-b482-6ed4210d1118-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-tmp\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-trusted-ca\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s66n\" (UniqueName: \"kubernetes.io/projected/f8be0c3f-c344-4968-92da-6e028a539e6f-kube-api-access-4s66n\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6e23da7-45bb-4993-ba2b-e82954181fbb-config\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-default-certificate\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-555j8\" (UniqueName: \"kubernetes.io/projected/a7589901-91ef-4eeb-8d9b-c5c60afb2368-kube-api-access-555j8\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.854834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fkbh\" (UniqueName: \"kubernetes.io/projected/75f039ba-8a29-4c46-8143-1b970f164e13-kube-api-access-2fkbh\") pod \"network-check-source-7bc5b6bf85-7n7z5\" (UID: \"75f039ba-8a29-4c46-8143-1b970f164e13\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854843    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854881    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-serving-cert\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e23da7-45bb-4993-ba2b-e82954181fbb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-tmp\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2987m\" (UniqueName: \"kubernetes.io/projected/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-kube-api-access-2987m\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.854982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwf8\" (UniqueName: \"kubernetes.io/projected/2001ea44-da85-4f05-b6a1-14552278aa80-kube-api-access-fkwf8\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p7zp\" (UniqueName: \"kubernetes.io/projected/d6e23da7-45bb-4993-ba2b-e82954181fbb-kube-api-access-2p7zp\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbsr\" (UniqueName: \"kubernetes.io/projected/cc2303f5-dba3-4879-969c-d31129b82416-kube-api-access-dxbsr\") pod \"volume-data-source-validator-78cf4c6f6-ntj9r\" (UID: \"cc2303f5-dba3-4879-969c-d31129b82416\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bae47ed-4f68-4b6f-b482-6ed4210d1118-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8be0c3f-c344-4968-92da-6e028a539e6f-config-volume\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.855230 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d6e23da7-45bb-4993-ba2b-e82954181fbb-config\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bae47ed-4f68-4b6f-b482-6ed4210d1118-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpnt\" (UniqueName: \"kubernetes.io/projected/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-kube-api-access-wmpnt\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855563    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855612    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-service-ca-bundle\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855633    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6h2t\" (UniqueName: \"kubernetes.io/projected/b4d66827-0a83-434c-b174-8224d6fa5830-kube-api-access-f6h2t\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8be0c3f-c344-4968-92da-6e028a539e6f-tmp-dir\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7589901-91ef-4eeb-8d9b-c5c60afb2368-cert\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-snapshots\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.855879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.855868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-stats-auth\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.856496 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.856368    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-snapshots\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.859273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.859249    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d6e23da7-45bb-4993-ba2b-e82954181fbb-serving-cert\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.859273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.859261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bae47ed-4f68-4b6f-b482-6ed4210d1118-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.859273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.859270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-serving-cert\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.863070 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.862988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jfp\" (UniqueName: \"kubernetes.io/projected/2bae47ed-4f68-4b6f-b482-6ed4210d1118-kube-api-access-c6jfp\") pod \"kube-storage-version-migrator-operator-7658d9b749-m87h2\" (UID: \"2bae47ed-4f68-4b6f-b482-6ed4210d1118\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:11.865192 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.865153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpnt\" (UniqueName: \"kubernetes.io/projected/954c5632-fd22-4e9b-9536-b8f6dd21b5c0-kube-api-access-wmpnt\") pod \"insights-operator-555d9d7885-cm2gp\" (UID: \"954c5632-fd22-4e9b-9536-b8f6dd21b5c0\") " pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.865312 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.865214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbsr\" (UniqueName: \"kubernetes.io/projected/cc2303f5-dba3-4879-969c-d31129b82416-kube-api-access-dxbsr\") pod \"volume-data-source-validator-78cf4c6f6-ntj9r\" (UID: \"cc2303f5-dba3-4879-969c-d31129b82416\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"
Jul 17 18:45:11.865312 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.865242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p7zp\" (UniqueName: \"kubernetes.io/projected/d6e23da7-45bb-4993-ba2b-e82954181fbb-kube-api-access-2p7zp\") pod \"service-ca-operator-5bcbc94c78-2g79l\" (UID: \"d6e23da7-45bb-4993-ba2b-e82954181fbb\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.956655 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-trusted-ca\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.956655 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s66n\" (UniqueName: \"kubernetes.io/projected/f8be0c3f-c344-4968-92da-6e028a539e6f-kube-api-access-4s66n\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-default-certificate\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-555j8\" (UniqueName: \"kubernetes.io/projected/a7589901-91ef-4eeb-8d9b-c5c60afb2368-kube-api-access-555j8\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fkbh\" (UniqueName: \"kubernetes.io/projected/75f039ba-8a29-4c46-8143-1b970f164e13-kube-api-access-2fkbh\") pod \"network-check-source-7bc5b6bf85-7n7z5\" (UID: \"75f039ba-8a29-4c46-8143-1b970f164e13\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.956853 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2987m\" (UniqueName: \"kubernetes.io/projected/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-kube-api-access-2987m\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.957015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwf8\" (UniqueName: \"kubernetes.io/projected/2001ea44-da85-4f05-b6a1-14552278aa80-kube-api-access-fkwf8\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.957015 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.956866    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:11.957015 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.956887    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:11.957015 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.956965    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.456942968 +0000 UTC m=+38.219373208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:11.957234 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.956885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.957308 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8be0c3f-c344-4968-92da-6e028a539e6f-config-volume\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.957308 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.957432 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.957432 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.957432 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.957579 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.957579 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.957493    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:11.957579 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.957539    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.457525151 +0000 UTC m=+38.219955389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:11.957579 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.957556    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.457547444 +0000 UTC m=+38.219977683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:11.957579 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6h2t\" (UniqueName: \"kubernetes.io/projected/b4d66827-0a83-434c-b174-8224d6fa5830-kube-api-access-f6h2t\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.957585    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8be0c3f-c344-4968-92da-6e028a539e6f-tmp-dir\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.957616    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.457604957 +0000 UTC m=+38.220035209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957723    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7589901-91ef-4eeb-8d9b-c5c60afb2368-cert\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-stats-auth\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.957833 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8be0c3f-c344-4968-92da-6e028a539e6f-tmp-dir\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjfn\" (UniqueName: \"kubernetes.io/projected/c4898006-61dc-42b4-a9d6-b3022d1adc4f-kube-api-access-wdjfn\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-config\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-trusted-ca\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.957991    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bz9\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.958024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d66827-0a83-434c-b174-8224d6fa5830-serving-cert\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.958022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f8be0c3f-c344-4968-92da-6e028a539e6f-config-volume\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.958115    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:11.958310 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.958172    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.458156552 +0000 UTC m=+38.220586794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:11.959040 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.958615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b4d66827-0a83-434c-b174-8224d6fa5830-config\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.959040 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.958791    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.960203 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.959252    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:11.960203 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:11.959348    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:12.459301031 +0000 UTC m=+38.221731279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:11.960203 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.960203 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-default-certificate\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.960203 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.960716 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960687    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7589901-91ef-4eeb-8d9b-c5c60afb2368-cert\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.960802 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b4d66827-0a83-434c-b174-8224d6fa5830-serving-cert\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.960924 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.960909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-stats-auth\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.961358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.961311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.965234 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.965220    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"
Jul 17 18:45:11.970372 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.970283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwf8\" (UniqueName: \"kubernetes.io/projected/2001ea44-da85-4f05-b6a1-14552278aa80-kube-api-access-fkwf8\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:11.970474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.970456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2987m\" (UniqueName: \"kubernetes.io/projected/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-kube-api-access-2987m\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:11.970991 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.970972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjfn\" (UniqueName: \"kubernetes.io/projected/c4898006-61dc-42b4-a9d6-b3022d1adc4f-kube-api-access-wdjfn\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:11.972442 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.972340    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-555j8\" (UniqueName: \"kubernetes.io/projected/a7589901-91ef-4eeb-8d9b-c5c60afb2368-kube-api-access-555j8\") pod \"ingress-canary-xpshk\" (UID: \"a7589901-91ef-4eeb-8d9b-c5c60afb2368\") " pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:11.973223 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.972765    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-cm2gp"
Jul 17 18:45:11.973223 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.973004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bz9\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.973223 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.973029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s66n\" (UniqueName: \"kubernetes.io/projected/f8be0c3f-c344-4968-92da-6e028a539e6f-kube-api-access-4s66n\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:11.974254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.974234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:11.974990 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.974957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fkbh\" (UniqueName: \"kubernetes.io/projected/75f039ba-8a29-4c46-8143-1b970f164e13-kube-api-access-2fkbh\") pod \"network-check-source-7bc5b6bf85-7n7z5\" (UID: \"75f039ba-8a29-4c46-8143-1b970f164e13\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"
Jul 17 18:45:11.975528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.975507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6h2t\" (UniqueName: \"kubernetes.io/projected/b4d66827-0a83-434c-b174-8224d6fa5830-kube-api-access-f6h2t\") pod \"console-operator-565b7b569d-wrcwt\" (UID: \"b4d66827-0a83-434c-b174-8224d6fa5830\") " pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:11.979997 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.979947    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"
Jul 17 18:45:11.986889 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:11.986873    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"
Jul 17 18:45:12.023574 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.023534    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:12.048564 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.038877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xpshk"
Jul 17 18:45:12.062811 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.062571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"
Jul 17 18:45:12.221599 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.221562    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r"]
Jul 17 18:45:12.237522 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.237076    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l"]
Jul 17 18:45:12.242015 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:12.241983    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e23da7_45bb_4993_ba2b_e82954181fbb.slice/crio-60176a1dc36d54b9cca4e7f9910f7a557e548644d1a5fcf3d5f110f24bdccc31 WatchSource:0}: Error finding container 60176a1dc36d54b9cca4e7f9910f7a557e548644d1a5fcf3d5f110f24bdccc31: Status 404 returned error can't find the container with id 60176a1dc36d54b9cca4e7f9910f7a557e548644d1a5fcf3d5f110f24bdccc31
Jul 17 18:45:12.252362 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.251937    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2"]
Jul 17 18:45:12.257737 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.256646    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-cm2gp"]
Jul 17 18:45:12.297400 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.297364    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-wrcwt"]
Jul 17 18:45:12.306637 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.306491    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xpshk"]
Jul 17 18:45:12.309289 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.309269    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5"]
Jul 17 18:45:12.465917 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.465836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:12.465917 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.465892    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.465964    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.465994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.466021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466032    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466094    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466108    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:12.466138 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466115    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466095    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466159    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466113    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466092442 +0000 UTC m=+39.228522684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:12.466042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466201    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466183799 +0000 UTC m=+39.228614048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466239    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466220514 +0000 UTC m=+39.228650759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466276    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466263248 +0000 UTC m=+39.228693492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466302    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466292847 +0000 UTC m=+39.228723085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:12.466434 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:12.466331    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:13.466311115 +0000 UTC m=+39.228741354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:13.007274 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.007239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l" event={"ID":"d6e23da7-45bb-4993-ba2b-e82954181fbb","Type":"ContainerStarted","Data":"60176a1dc36d54b9cca4e7f9910f7a557e548644d1a5fcf3d5f110f24bdccc31"}
Jul 17 18:45:13.008492 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.008457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r" event={"ID":"cc2303f5-dba3-4879-969c-d31129b82416","Type":"ContainerStarted","Data":"8c25a9549216fa7d324545d12b37c6524db937d1481fd7b7cc878ec5d23747c3"}
Jul 17 18:45:13.475817 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.475766    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.475860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.475932    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.475944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.475975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.475986    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476001    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476010    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.475989389 +0000 UTC m=+41.238419649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:13.476031 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476035    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.476025488 +0000 UTC m=+41.238455729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.476035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:13.476072    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476076    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476083    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476127    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.476112012 +0000 UTC m=+41.238542264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476146    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.476134963 +0000 UTC m=+41.238565205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476164    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476181    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.476167329 +0000 UTC m=+41.238597568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:13.476419 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:13.476201    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:15.47619119 +0000 UTC m=+41.238621435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:13.838472 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:13.838434    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bae47ed_4f68_4b6f_b482_6ed4210d1118.slice/crio-b0002f8a95aca97c4499fbf5a3a9d2340609d300b534f7d1571ba7f03adb6c2b WatchSource:0}: Error finding container b0002f8a95aca97c4499fbf5a3a9d2340609d300b534f7d1571ba7f03adb6c2b: Status 404 returned error can't find the container with id b0002f8a95aca97c4499fbf5a3a9d2340609d300b534f7d1571ba7f03adb6c2b
Jul 17 18:45:13.839877 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:13.839848    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d66827_0a83_434c_b174_8224d6fa5830.slice/crio-b416e7216e3750e5b8f1c89e48253d81ee847b4a1f0af8a1570057e78b52eb10 WatchSource:0}: Error finding container b416e7216e3750e5b8f1c89e48253d81ee847b4a1f0af8a1570057e78b52eb10: Status 404 returned error can't find the container with id b416e7216e3750e5b8f1c89e48253d81ee847b4a1f0af8a1570057e78b52eb10
Jul 17 18:45:13.840506 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:13.840476    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7589901_91ef_4eeb_8d9b_c5c60afb2368.slice/crio-dc12f94fdff38fa94ffc41e2100fc56fec332a563c8ef7d5d3f2cda79d46d30a WatchSource:0}: Error finding container dc12f94fdff38fa94ffc41e2100fc56fec332a563c8ef7d5d3f2cda79d46d30a: Status 404 returned error can't find the container with id dc12f94fdff38fa94ffc41e2100fc56fec332a563c8ef7d5d3f2cda79d46d30a
Jul 17 18:45:13.842859 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:13.842388    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954c5632_fd22_4e9b_9536_b8f6dd21b5c0.slice/crio-038c425baad17f8877e23b21d5fc970f924de53408c93f925fdb01943039fd59 WatchSource:0}: Error finding container 038c425baad17f8877e23b21d5fc970f924de53408c93f925fdb01943039fd59: Status 404 returned error can't find the container with id 038c425baad17f8877e23b21d5fc970f924de53408c93f925fdb01943039fd59
Jul 17 18:45:13.843911 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:13.843161    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f039ba_8a29_4c46_8143_1b970f164e13.slice/crio-d2db0d76bbd39f5e9871a5f1dd8a1f2e3a6ffc563ef0b3815c371593a959639f WatchSource:0}: Error finding container d2db0d76bbd39f5e9871a5f1dd8a1f2e3a6ffc563ef0b3815c371593a959639f: Status 404 returned error can't find the container with id d2db0d76bbd39f5e9871a5f1dd8a1f2e3a6ffc563ef0b3815c371593a959639f
Jul 17 18:45:14.011766 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:14.011737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" event={"ID":"b4d66827-0a83-434c-b174-8224d6fa5830","Type":"ContainerStarted","Data":"b416e7216e3750e5b8f1c89e48253d81ee847b4a1f0af8a1570057e78b52eb10"}
Jul 17 18:45:14.012764 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:14.012739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cm2gp" event={"ID":"954c5632-fd22-4e9b-9536-b8f6dd21b5c0","Type":"ContainerStarted","Data":"038c425baad17f8877e23b21d5fc970f924de53408c93f925fdb01943039fd59"}
Jul 17 18:45:14.013698 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:14.013679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2" event={"ID":"2bae47ed-4f68-4b6f-b482-6ed4210d1118","Type":"ContainerStarted","Data":"b0002f8a95aca97c4499fbf5a3a9d2340609d300b534f7d1571ba7f03adb6c2b"}
Jul 17 18:45:14.014635 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:14.014611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5" event={"ID":"75f039ba-8a29-4c46-8143-1b970f164e13","Type":"ContainerStarted","Data":"d2db0d76bbd39f5e9871a5f1dd8a1f2e3a6ffc563ef0b3815c371593a959639f"}
Jul 17 18:45:14.015526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:14.015508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpshk" event={"ID":"a7589901-91ef-4eeb-8d9b-c5c60afb2368","Type":"ContainerStarted","Data":"dc12f94fdff38fa94ffc41e2100fc56fec332a563c8ef7d5d3f2cda79d46d30a"}
Jul 17 18:45:15.030390 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.029755    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="5e6d97fe264675f80b4bb04fd68aa5521c2b19970f7c9cc990ab96a86f4a2d98" exitCode=0
Jul 17 18:45:15.030390 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.030032    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"5e6d97fe264675f80b4bb04fd68aa5521c2b19970f7c9cc990ab96a86f4a2d98"}
Jul 17 18:45:15.496549 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:15.496726 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:15.496726 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:15.496726 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:15.496726 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:15.496936 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.496756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:15.496936 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.496896    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:15.497022 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.496961    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.496941992 +0000 UTC m=+45.259372243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:15.497466 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497441    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:15.497588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497499    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.497482361 +0000 UTC m=+45.259912600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:15.497588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497561    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:15.497588 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497572    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:15.497769 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497605    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.497594248 +0000 UTC m=+45.260024491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:15.497769 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497657    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:15.497769 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497692    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.497681126 +0000 UTC m=+45.260111365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:15.497769 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497759    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.497749139 +0000 UTC m=+45.260179380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:15.497979 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497865    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:15.497979 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.497901    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.497890363 +0000 UTC m=+45.260320605 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:15.655339 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.655294    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"]
Jul 17 18:45:15.666874 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.666844    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:15.669856 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.669797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"]
Jul 17 18:45:15.671919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.671256    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fhdlj\""
Jul 17 18:45:15.671919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.671521    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 17 18:45:15.671919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.671754    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 17 18:45:15.700680 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.699739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:15.700680 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.699778    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:15.800877 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.800824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:15.800877 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.800878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:15.801146 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.801130    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:15.801208 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:15.801195    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:16.301175849 +0000 UTC m=+42.063606103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:15.802134 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:15.802092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:16.043220 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:16.042188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerStarted","Data":"87b73b6f805c8a4c4f77a3168d680b47886269058d1b82fb66be5f1f3e344978"}
Jul 17 18:45:16.305275 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:16.305236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:16.305528 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:16.305509    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:16.305591 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:16.305581    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:17.305563544 +0000 UTC m=+43.067993788 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:17.048725 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:17.048641    2558 generic.go:356] "Generic (PLEG): container finished" podID="0572a173-ab6b-43d2-90ab-522059991d5c" containerID="87b73b6f805c8a4c4f77a3168d680b47886269058d1b82fb66be5f1f3e344978" exitCode=0
Jul 17 18:45:17.049157 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:17.048736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerDied","Data":"87b73b6f805c8a4c4f77a3168d680b47886269058d1b82fb66be5f1f3e344978"}
Jul 17 18:45:17.315152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:17.315120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:17.315295 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:17.315271    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:17.315371 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:17.315355    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:19.315337315 +0000 UTC m=+45.077767579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:19.335041 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.335000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:19.335542 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.335160    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:19.335542 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.335239    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:23.335222372 +0000 UTC m=+49.097652609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:19.536467 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:19.536636 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:19.536636 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536593    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536659    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536642307 +0000 UTC m=+53.299072545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536670    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536674    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536745    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:19.536753 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536781    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536765603 +0000 UTC m=+53.299195871 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536800    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536808    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:19.536836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536846    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536833698 +0000 UTC m=+53.299263939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536879    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536863321 +0000 UTC m=+53.299293559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536892    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536921    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536909884 +0000 UTC m=+53.299340123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:19.537043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:19.536963    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.536946858 +0000 UTC m=+53.299377096 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:22.063242 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.063207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l" event={"ID":"d6e23da7-45bb-4993-ba2b-e82954181fbb","Type":"ContainerStarted","Data":"a2b6098a1b18b56fd485eef65e943b60311a267c2156b7b81ed69b35465c17eb"}
Jul 17 18:45:22.065349 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.065272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2" event={"ID":"2bae47ed-4f68-4b6f-b482-6ed4210d1118","Type":"ContainerStarted","Data":"6da3bc2dfc4f77b95561550d00f294c4dd2aecae2a4957cc669c33329192166a"}
Jul 17 18:45:22.066749 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.066722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5" event={"ID":"75f039ba-8a29-4c46-8143-1b970f164e13","Type":"ContainerStarted","Data":"ab3ad819edb91cf96f70bcf6426fe5103b8c71c97fba2d81f322ca3796250751"}
Jul 17 18:45:22.068341 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.068297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xpshk" event={"ID":"a7589901-91ef-4eeb-8d9b-c5c60afb2368","Type":"ContainerStarted","Data":"a6627037cccb93b57668496489d68ce550adf636aa3ed996c5dd7308f8dcd8bd"}
Jul 17 18:45:22.070063 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.070046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r" event={"ID":"cc2303f5-dba3-4879-969c-d31129b82416","Type":"ContainerStarted","Data":"8a50c615715daf2b92c6a9d920a71db22642d067bc1f511cd2f41e132082dfd4"}
Jul 17 18:45:22.071907 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.071891    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/0.log"
Jul 17 18:45:22.071990 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.071916    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4d66827-0a83-434c-b174-8224d6fa5830" containerID="e68ab363c72b1ec4d5b21044e1c9f1add801f6891138e3aae7adcf600de9c479" exitCode=255
Jul 17 18:45:22.071990 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.071965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" event={"ID":"b4d66827-0a83-434c-b174-8224d6fa5830","Type":"ContainerDied","Data":"e68ab363c72b1ec4d5b21044e1c9f1add801f6891138e3aae7adcf600de9c479"}
Jul 17 18:45:22.072154 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.072141    2558 scope.go:117] "RemoveContainer" containerID="e68ab363c72b1ec4d5b21044e1c9f1add801f6891138e3aae7adcf600de9c479"
Jul 17 18:45:22.074222 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.074201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cm2gp" event={"ID":"954c5632-fd22-4e9b-9536-b8f6dd21b5c0","Type":"ContainerStarted","Data":"eba0698d9958d75e36eff669fbc1540cdd0e8bba355cd38975c5b86159deac58"}
Jul 17 18:45:22.078143 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.078122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zztfp" event={"ID":"0572a173-ab6b-43d2-90ab-522059991d5c","Type":"ContainerStarted","Data":"ade72fc9edba54c65594df4178a381627ed3e5c3702049a4dc06a69c59ed7a4b"}
Jul 17 18:45:22.120902 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.120776    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zztfp" podStartSLOduration=21.71741151 podStartE2EDuration="47.120754954s" totalImagesPullingTime="25.403343444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.120083936 +0000 UTC m=+47.882514197" watchObservedRunningTime="2026-07-17 18:45:22.120754954 +0000 UTC m=+47.883185217"
Jul 17 18:45:22.121342 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.121291    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l" podStartSLOduration=24.851700751 podStartE2EDuration="34.121275665s" totalImagesPullingTime="9.269574914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.09115962 +0000 UTC m=+47.853589880" watchObservedRunningTime="2026-07-17 18:45:22.121275665 +0000 UTC m=+47.883705926"
Jul 17 18:45:22.173342 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.173274    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-7n7z5" podStartSLOduration=26.490699137 podStartE2EDuration="34.173252907s" totalImagesPullingTime="7.68255377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.173074219 +0000 UTC m=+47.935504482" watchObservedRunningTime="2026-07-17 18:45:22.173252907 +0000 UTC m=+47.935683169"
Jul 17 18:45:22.174547 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.174504    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xpshk" podStartSLOduration=3.5016072129999998 podStartE2EDuration="11.174489518s" totalImagesPullingTime="7.672882305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.14352627 +0000 UTC m=+47.905956531" watchObservedRunningTime="2026-07-17 18:45:22.174489518 +0000 UTC m=+47.936919774"
Jul 17 18:45:22.247390 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.247335    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2" podStartSLOduration=26.571519046 podStartE2EDuration="34.247297756s" totalImagesPullingTime="7.67577871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.246695718 +0000 UTC m=+48.009125979" watchObservedRunningTime="2026-07-17 18:45:22.247297756 +0000 UTC m=+48.009728018"
Jul 17 18:45:22.273406 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.273059    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-ntj9r" podStartSLOduration=25.456245565 podStartE2EDuration="34.273038915s" totalImagesPullingTime="8.81679335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.272347031 +0000 UTC m=+48.034777317" watchObservedRunningTime="2026-07-17 18:45:22.273038915 +0000 UTC m=+48.035469177"
Jul 17 18:45:22.330817 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:22.330539    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-cm2gp" podStartSLOduration=26.658182277 podStartE2EDuration="34.330517417s" totalImagesPullingTime="7.67233514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.329253246 +0000 UTC m=+48.091683508" watchObservedRunningTime="2026-07-17 18:45:22.330517417 +0000 UTC m=+48.092947679"
Jul 17 18:45:23.084450 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.084410    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:45:23.084902 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.084844    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/0.log"
Jul 17 18:45:23.084902 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.084882    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4d66827-0a83-434c-b174-8224d6fa5830" containerID="03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0" exitCode=255
Jul 17 18:45:23.085015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.084981    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" event={"ID":"b4d66827-0a83-434c-b174-8224d6fa5830","Type":"ContainerDied","Data":"03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0"}
Jul 17 18:45:23.085066 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.085049    2558 scope.go:117] "RemoveContainer" containerID="e68ab363c72b1ec4d5b21044e1c9f1add801f6891138e3aae7adcf600de9c479"
Jul 17 18:45:23.085403 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.085383    2558 scope.go:117] "RemoveContainer" containerID="03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0"
Jul 17 18:45:23.085593 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.085571    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wrcwt_openshift-console-operator(b4d66827-0a83-434c-b174-8224d6fa5830)\"" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" podUID="b4d66827-0a83-434c-b174-8224d6fa5830"
Jul 17 18:45:23.379195 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.379101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:23.379360 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.379278    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:23.379426 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.379384    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:31.379360606 +0000 UTC m=+57.141790856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:23.449243 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.449208    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6qkzh"]
Jul 17 18:45:23.482264 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.482233    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qkzh"]
Jul 17 18:45:23.482424 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.482352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.485475 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.485454    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 18:45:23.485475 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.485473    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-htzm6\""
Jul 17 18:45:23.485757 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.485742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 18:45:23.580640 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.580604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ldv\" (UniqueName: \"kubernetes.io/projected/87336d0b-4178-4b44-b999-a44ed8434de8-kube-api-access-h9ldv\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.580814 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.580684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87336d0b-4178-4b44-b999-a44ed8434de8-data-volume\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.580814 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.580714    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87336d0b-4178-4b44-b999-a44ed8434de8-crio-socket\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.580814 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.580734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.580981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.580882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682332 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87336d0b-4178-4b44-b999-a44ed8434de8-data-volume\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682332 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87336d0b-4178-4b44-b999-a44ed8434de8-crio-socket\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682332 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/87336d0b-4178-4b44-b999-a44ed8434de8-crio-socket\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682603 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.682421    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:45:23.682603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682603 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.682529    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:24.18251169 +0000 UTC m=+49.944941931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : configmap "kube-rbac-proxy" not found
Jul 17 18:45:23.682603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ldv\" (UniqueName: \"kubernetes.io/projected/87336d0b-4178-4b44-b999-a44ed8434de8-kube-api-access-h9ldv\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682819 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.682617    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:23.682819 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.682649    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/87336d0b-4178-4b44-b999-a44ed8434de8-data-volume\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:23.682819 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:23.682661    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:24.182648714 +0000 UTC m=+49.945078955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:23.698830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:23.698801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ldv\" (UniqueName: \"kubernetes.io/projected/87336d0b-4178-4b44-b999-a44ed8434de8-kube-api-access-h9ldv\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:24.089706 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:24.089682    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:45:24.090081 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:24.089991    2558 scope.go:117] "RemoveContainer" containerID="03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0"
Jul 17 18:45:24.090167 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:24.090143    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wrcwt_openshift-console-operator(b4d66827-0a83-434c-b174-8224d6fa5830)\"" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" podUID="b4d66827-0a83-434c-b174-8224d6fa5830"
Jul 17 18:45:24.186815 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:24.186784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:24.186997 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:24.186859    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:45:24.186997 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:24.186920    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:25.186906398 +0000 UTC m=+50.949336636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : configmap "kube-rbac-proxy" not found
Jul 17 18:45:24.186997 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:24.186941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:24.187159 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:24.187129    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:24.187208 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:24.187185    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:25.187169184 +0000 UTC m=+50.949599429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:25.195058 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:25.195024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:25.195455 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:25.195094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:25.195455 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:25.195145    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 18:45:25.195455 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:25.195208    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.195192177 +0000 UTC m=+52.957622414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : configmap "kube-rbac-proxy" not found
Jul 17 18:45:25.195455 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:25.195212    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:25.195455 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:25.195250    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:27.195239658 +0000 UTC m=+52.957669896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:25.237101 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:25.237068    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcbkz_24ba022f-d700-4dea-83fe-3f00bd2bd1db/dns-node-resolver/0.log"
Jul 17 18:45:25.819037 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:25.819010    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htt78_f4c47689-ae80-4e61-bb80-f4cb8f3dfadf/node-ca/0.log"
Jul 17 18:45:26.830224 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:26.830192    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpshk_a7589901-91ef-4eeb-8d9b-c5c60afb2368/serve-healthcheck-canary/0.log"
Jul 17 18:45:27.213756 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.213671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:27.213928 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.213762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:27.213990 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.213929    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:27.214043 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.214022    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:31.214001111 +0000 UTC m=+56.976431370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:27.214211 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.214190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/87336d0b-4178-4b44-b999-a44ed8434de8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:27.617721 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617803    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617851    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617877    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c777bf7d5-pln4f: secret "image-registry-tls" not found
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:27.617908    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617929    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617931    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 17 18:45:27.618007 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617984    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.617995    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls podName:42b75940-c085-4f8c-bce5-4d23c1f6a7b2 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.617972778 +0000 UTC m=+69.380403039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-b4pxl" (UID: "42b75940-c085-4f8c-bce5-4d23c1f6a7b2") : secret "cluster-monitoring-operator-tls" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618068    2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618089    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls podName:1f0543a3-7342-40cc-a87d-c261298366c4 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.618072524 +0000 UTC m=+69.380502766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls") pod "image-registry-6c777bf7d5-pln4f" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4") : secret "image-registry-tls" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618102    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.618096068 +0000 UTC m=+69.380526307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : configmap references non-existent config key: service-ca.crt
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618112    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls podName:2001ea44-da85-4f05-b6a1-14552278aa80 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.618106732 +0000 UTC m=+69.380536970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-sms5j" (UID: "2001ea44-da85-4f05-b6a1-14552278aa80") : secret "samples-operator-tls" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618122    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls podName:f8be0c3f-c344-4968-92da-6e028a539e6f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.618117224 +0000 UTC m=+69.380547462 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls") pod "dns-default-hph6v" (UID: "f8be0c3f-c344-4968-92da-6e028a539e6f") : secret "dns-default-metrics-tls" not found
Jul 17 18:45:27.618354 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:27.618133    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs podName:c4898006-61dc-42b4-a9d6-b3022d1adc4f nodeName:}" failed. No retries permitted until 2026-07-17 18:45:43.618128868 +0000 UTC m=+69.380559106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs") pod "router-default-5c6cb4dc5b-f5hkm" (UID: "c4898006-61dc-42b4-a9d6-b3022d1adc4f") : secret "router-metrics-certs-default" not found
Jul 17 18:45:31.251564 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:31.251530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:31.251933 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:31.251680    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:31.251933 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:31.251744    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls podName:87336d0b-4178-4b44-b999-a44ed8434de8 nodeName:}" failed. No retries permitted until 2026-07-17 18:45:39.251727339 +0000 UTC m=+65.014157602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qkzh" (UID: "87336d0b-4178-4b44-b999-a44ed8434de8") : secret "insights-runtime-extractor-tls" not found
Jul 17 18:45:31.453644 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:31.453604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:31.453802 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:31.453781    2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 17 18:45:31.453860 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:31.453850    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert podName:d22bd50b-1802-473b-8dd7-1e5a96e7ff6b nodeName:}" failed. No retries permitted until 2026-07-17 18:45:47.453835038 +0000 UTC m=+73.216265275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-pp97k" (UID: "d22bd50b-1802-473b-8dd7-1e5a96e7ff6b") : secret "networking-console-plugin-cert" not found
Jul 17 18:45:32.024094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:32.024056    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:32.024267 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:32.024119    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:32.024523 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:32.024510    2558 scope.go:117] "RemoveContainer" containerID="03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0"
Jul 17 18:45:32.024676 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:45:32.024656    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-wrcwt_openshift-console-operator(b4d66827-0a83-434c-b174-8224d6fa5830)\"" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" podUID="b4d66827-0a83-434c-b174-8224d6fa5830"
Jul 17 18:45:37.002645 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:37.002619    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd5jh"
Jul 17 18:45:39.328630 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:39.328596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:39.330980 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:39.330963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/87336d0b-4178-4b44-b999-a44ed8434de8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qkzh\" (UID: \"87336d0b-4178-4b44-b999-a44ed8434de8\") " pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:39.394426 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:39.394389    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-htzm6\""
Jul 17 18:45:39.401633 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:39.401607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qkzh"
Jul 17 18:45:39.529284 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:39.529255    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qkzh"]
Jul 17 18:45:39.532311 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:39.532292    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87336d0b_4178_4b44_b999_a44ed8434de8.slice/crio-a2044eb93664fe20501a269cf8bd08b23ef62dae026784f9e5b845d7c98e4a8a WatchSource:0}: Error finding container a2044eb93664fe20501a269cf8bd08b23ef62dae026784f9e5b845d7c98e4a8a: Status 404 returned error can't find the container with id a2044eb93664fe20501a269cf8bd08b23ef62dae026784f9e5b845d7c98e4a8a
Jul 17 18:45:40.140295 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.140251    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qkzh" event={"ID":"87336d0b-4178-4b44-b999-a44ed8434de8","Type":"ContainerStarted","Data":"a2044eb93664fe20501a269cf8bd08b23ef62dae026784f9e5b845d7c98e4a8a"}
Jul 17 18:45:40.439231 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.439152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:40.439231 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.439220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:40.442155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.441957    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 18:45:40.442860 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.442839    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 18:45:40.456701 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.456670    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef151684-97e8-459b-aec6-07672b896efc-metrics-certs\") pod \"network-metrics-daemon-hb6vp\" (UID: \"ef151684-97e8-459b-aec6-07672b896efc\") " pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:40.456815 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.456673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3761999b-61c5-40c7-8d95-5c228ce395b0-original-pull-secret\") pod \"global-pull-secret-syncer-wv5n9\" (UID: \"3761999b-61c5-40c7-8d95-5c228ce395b0\") " pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:40.579932 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.579882    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wv5n9"
Jul 17 18:45:40.592446 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.592421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jqp7v\""
Jul 17 18:45:40.599823 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.599606    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hb6vp"
Jul 17 18:45:40.642087 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.641505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:40.652039 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.651974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4f92\" (UniqueName: \"kubernetes.io/projected/d0b8270f-1579-4924-9d71-8aeb28490f59-kube-api-access-d4f92\") pod \"network-check-target-sqfrw\" (UID: \"d0b8270f-1579-4924-9d71-8aeb28490f59\") " pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:40.744696 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.744545    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wv5n9"]
Jul 17 18:45:40.747365 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:40.747314    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3761999b_61c5_40c7_8d95_5c228ce395b0.slice/crio-d8b65d9ecf19ee35d033e84144277bd4d1243a434608c7d3aa2272ede7ab5e4c WatchSource:0}: Error finding container d8b65d9ecf19ee35d033e84144277bd4d1243a434608c7d3aa2272ede7ab5e4c: Status 404 returned error can't find the container with id d8b65d9ecf19ee35d033e84144277bd4d1243a434608c7d3aa2272ede7ab5e4c
Jul 17 18:45:40.760224 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.760193    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hb6vp"]
Jul 17 18:45:40.762487 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:40.762461    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef151684_97e8_459b_aec6_07672b896efc.slice/crio-8c27e04b6570584bb429a5c285d2176046144265cef688b53a50e4aea5eec7bd WatchSource:0}: Error finding container 8c27e04b6570584bb429a5c285d2176046144265cef688b53a50e4aea5eec7bd: Status 404 returned error can't find the container with id 8c27e04b6570584bb429a5c285d2176046144265cef688b53a50e4aea5eec7bd
Jul 17 18:45:40.896423 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.896390    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ktbms\""
Jul 17 18:45:40.904738 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:40.904714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:41.047444 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.047380    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sqfrw"]
Jul 17 18:45:41.050498 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:41.050467    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b8270f_1579_4924_9d71_8aeb28490f59.slice/crio-194fb8e558f9516a91c241a6a754a331a74995e8ca1dc3594e13df4766cde1c0 WatchSource:0}: Error finding container 194fb8e558f9516a91c241a6a754a331a74995e8ca1dc3594e13df4766cde1c0: Status 404 returned error can't find the container with id 194fb8e558f9516a91c241a6a754a331a74995e8ca1dc3594e13df4766cde1c0
Jul 17 18:45:41.145518 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.145486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sqfrw" event={"ID":"d0b8270f-1579-4924-9d71-8aeb28490f59","Type":"ContainerStarted","Data":"c214eb4a49b7938132f3515a4f2c5ac124b15d4db21bbdfb57cfa5d50c9e6bd0"}
Jul 17 18:45:41.145657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.145542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sqfrw" event={"ID":"d0b8270f-1579-4924-9d71-8aeb28490f59","Type":"ContainerStarted","Data":"194fb8e558f9516a91c241a6a754a331a74995e8ca1dc3594e13df4766cde1c0"}
Jul 17 18:45:41.145657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.145571    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:45:41.146935 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.146892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wv5n9" event={"ID":"3761999b-61c5-40c7-8d95-5c228ce395b0","Type":"ContainerStarted","Data":"d8b65d9ecf19ee35d033e84144277bd4d1243a434608c7d3aa2272ede7ab5e4c"}
Jul 17 18:45:41.148208 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.148177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6vp" event={"ID":"ef151684-97e8-459b-aec6-07672b896efc","Type":"ContainerStarted","Data":"8c27e04b6570584bb429a5c285d2176046144265cef688b53a50e4aea5eec7bd"}
Jul 17 18:45:41.170432 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:41.170385    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sqfrw" podStartSLOduration=66.170371385 podStartE2EDuration="1m6.170371385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:41.169338165 +0000 UTC m=+66.931768416" watchObservedRunningTime="2026-07-17 18:45:41.170371385 +0000 UTC m=+66.932801644"
Jul 17 18:45:43.669843 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.669806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:43.670246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.669861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:43.670246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.669911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:43.670246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.669982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:43.670246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.670022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:43.670246 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.670040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:43.670644 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.670613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4898006-61dc-42b4-a9d6-b3022d1adc4f-service-ca-bundle\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:43.673385 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.673310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"image-registry-6c777bf7d5-pln4f\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") " pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:43.673502 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.673455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4898006-61dc-42b4-a9d6-b3022d1adc4f-metrics-certs\") pod \"router-default-5c6cb4dc5b-f5hkm\" (UID: \"c4898006-61dc-42b4-a9d6-b3022d1adc4f\") " pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:43.673502 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.673457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/42b75940-c085-4f8c-bce5-4d23c1f6a7b2-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-b4pxl\" (UID: \"42b75940-c085-4f8c-bce5-4d23c1f6a7b2\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:43.673781 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.673752    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8be0c3f-c344-4968-92da-6e028a539e6f-metrics-tls\") pod \"dns-default-hph6v\" (UID: \"f8be0c3f-c344-4968-92da-6e028a539e6f\") " pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:43.675555 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.675524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2001ea44-da85-4f05-b6a1-14552278aa80-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-sms5j\" (UID: \"2001ea44-da85-4f05-b6a1-14552278aa80\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:43.806282 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.806246    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-czwj6\""
Jul 17 18:45:43.813806 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.813776    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"
Jul 17 18:45:43.815748 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.815728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kmznd\""
Jul 17 18:45:43.823886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.823817    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:43.833005 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.832984    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f8z7z\""
Jul 17 18:45:43.841090 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.841067    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"
Jul 17 18:45:43.848098 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.848075    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4m4h7\""
Jul 17 18:45:43.855766 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.855478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:43.858264 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.858245    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zt9vq\""
Jul 17 18:45:43.866656 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:43.866591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:44.769009 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:44.768979    2558 scope.go:117] "RemoveContainer" containerID="03d89af295b518404fd513dcf0726cb5635511aa446682642627df98fc92b9d0"
Jul 17 18:45:46.025410 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.025373    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:45:46.126179 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.126149    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84fb5db76b-f97n9"]
Jul 17 18:45:46.130698 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.130674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.155102 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.154827    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb5db76b-f97n9"]
Jul 17 18:45:46.291993 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.291906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-trusted-ca\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.291993 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.291958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-installation-pull-secrets\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292187 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-bound-sa-token\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292187 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292090    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a883a90-b8e3-46eb-a99d-eeded91debbd-ca-trust-extracted\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292187 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292163    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-image-registry-private-configuration\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292187 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292183    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7q66\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-kube-api-access-b7q66\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-certificates\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.292358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.292222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-tls\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393126 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-image-registry-private-configuration\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393283 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7q66\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-kube-api-access-b7q66\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393283 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-certificates\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393283 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-tls\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393283 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-trusted-ca\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393518 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-installation-pull-secrets\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393518 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-bound-sa-token\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.393518 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.393395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a883a90-b8e3-46eb-a99d-eeded91debbd-ca-trust-extracted\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.394052 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.394023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0a883a90-b8e3-46eb-a99d-eeded91debbd-ca-trust-extracted\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.394301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.394272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-certificates\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.394414 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.394294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0a883a90-b8e3-46eb-a99d-eeded91debbd-trusted-ca\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.396191 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.396171    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-installation-pull-secrets\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.396362 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.396341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0a883a90-b8e3-46eb-a99d-eeded91debbd-image-registry-private-configuration\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.396417 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.396398    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-registry-tls\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.416147 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.416112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7q66\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-kube-api-access-b7q66\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.416295 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.416244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a883a90-b8e3-46eb-a99d-eeded91debbd-bound-sa-token\") pod \"image-registry-84fb5db76b-f97n9\" (UID: \"0a883a90-b8e3-46eb-a99d-eeded91debbd\") " pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:46.442124 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:46.442092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:47.019962 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.019865    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hph6v"]
Jul 17 18:45:47.023862 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:47.023833    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8be0c3f_c344_4968_92da_6e028a539e6f.slice/crio-8c9c767ee8b38fdb6f82d4b29e28e270ed17eea69381c9d6d60993e2054ca4b7 WatchSource:0}: Error finding container 8c9c767ee8b38fdb6f82d4b29e28e270ed17eea69381c9d6d60993e2054ca4b7: Status 404 returned error can't find the container with id 8c9c767ee8b38fdb6f82d4b29e28e270ed17eea69381c9d6d60993e2054ca4b7
Jul 17 18:45:47.108516 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.108485    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:45:47.112524 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:47.112493    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0543a3_7342_40cc_a87d_c261298366c4.slice/crio-ac688228fc9bd044e060decef9018718e9c9bd232dac6dbe3068a6fe7c21a9cb WatchSource:0}: Error finding container ac688228fc9bd044e060decef9018718e9c9bd232dac6dbe3068a6fe7c21a9cb: Status 404 returned error can't find the container with id ac688228fc9bd044e060decef9018718e9c9bd232dac6dbe3068a6fe7c21a9cb
Jul 17 18:45:47.172236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.172216    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:45:47.172378 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.172356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" event={"ID":"b4d66827-0a83-434c-b174-8224d6fa5830","Type":"ContainerStarted","Data":"84f9497a543cffe133bcaaae9c94cf8560abc1b8c7d4cd0425fd8840a1ec7b9b"}
Jul 17 18:45:47.173091 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.173009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:47.174505 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.174479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wv5n9" event={"ID":"3761999b-61c5-40c7-8d95-5c228ce395b0","Type":"ContainerStarted","Data":"7af51043082090c3e98d9f3115b352feea74666cb4d38a99a44115e35496b086"}
Jul 17 18:45:47.175656 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.175635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hph6v" event={"ID":"f8be0c3f-c344-4968-92da-6e028a539e6f","Type":"ContainerStarted","Data":"8c9c767ee8b38fdb6f82d4b29e28e270ed17eea69381c9d6d60993e2054ca4b7"}
Jul 17 18:45:47.177627 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.177607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6vp" event={"ID":"ef151684-97e8-459b-aec6-07672b896efc","Type":"ContainerStarted","Data":"cc8e903a840f77e60c1dab34e0471331b3dcc4ce98465c7b75c641e254a2d1ca"}
Jul 17 18:45:47.177701 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.177642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hb6vp" event={"ID":"ef151684-97e8-459b-aec6-07672b896efc","Type":"ContainerStarted","Data":"9f449c42b63db5542c2acaf81b1867d2b1e69b9d0876754e389366d1680f8155"}
Jul 17 18:45:47.179933 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.179910    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qkzh" event={"ID":"87336d0b-4178-4b44-b999-a44ed8434de8","Type":"ContainerStarted","Data":"ae991c4aa1084de02ffe3487831d21830b2684c00cb50608a74f5b4101e86f49"}
Jul 17 18:45:47.181342 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.181298    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" event={"ID":"1f0543a3-7342-40cc-a87d-c261298366c4","Type":"ContainerStarted","Data":"ac688228fc9bd044e060decef9018718e9c9bd232dac6dbe3068a6fe7c21a9cb"}
Jul 17 18:45:47.181477 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.181463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:45:47.195271 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.195084    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt" podStartSLOduration=51.521894706 podStartE2EDuration="59.195065114s" totalImagesPullingTime="7.673170408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:47.193958349 +0000 UTC m=+72.956388631" watchObservedRunningTime="2026-07-17 18:45:47.195065114 +0000 UTC m=+72.957495374"
Jul 17 18:45:47.256299 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.256182    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hb6vp" podStartSLOduration=66.297186689 podStartE2EDuration="1m12.256161539s" totalImagesPullingTime="5.95897485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:47.223928593 +0000 UTC m=+72.986358855" watchObservedRunningTime="2026-07-17 18:45:47.256161539 +0000 UTC m=+73.018591800"
Jul 17 18:45:47.257033 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.257012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl"]
Jul 17 18:45:47.260810 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.260777    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j"]
Jul 17 18:45:47.261294 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:47.261267    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b75940_c085_4f8c_bce5_4d23c1f6a7b2.slice/crio-1fd824ef8e7ec760b061c6682eb39cd123b5dee183970651cd8b4a3368f321b4 WatchSource:0}: Error finding container 1fd824ef8e7ec760b061c6682eb39cd123b5dee183970651cd8b4a3368f321b4: Status 404 returned error can't find the container with id 1fd824ef8e7ec760b061c6682eb39cd123b5dee183970651cd8b4a3368f321b4
Jul 17 18:45:47.268012 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.267592    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84fb5db76b-f97n9"]
Jul 17 18:45:47.272082 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.272020    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5c6cb4dc5b-f5hkm"]
Jul 17 18:45:47.275754 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.272194    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" podStartSLOduration=59.272171135 podStartE2EDuration="59.272171135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:47.262660518 +0000 UTC m=+73.025090779" watchObservedRunningTime="2026-07-17 18:45:47.272171135 +0000 UTC m=+73.034601394"
Jul 17 18:45:47.280664 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:45:47.280638    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a883a90_b8e3_46eb_a99d_eeded91debbd.slice/crio-3427f77942b13ea0089f6da7afdad8cf50f0a76d39790052c794a1baa7223c7c WatchSource:0}: Error finding container 3427f77942b13ea0089f6da7afdad8cf50f0a76d39790052c794a1baa7223c7c: Status 404 returned error can't find the container with id 3427f77942b13ea0089f6da7afdad8cf50f0a76d39790052c794a1baa7223c7c
Jul 17 18:45:47.290970 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.290911    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wv5n9" podStartSLOduration=66.26930075 podStartE2EDuration="1m12.290893193s" totalImagesPullingTime="6.021592443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:47.290392346 +0000 UTC m=+73.052822607" watchObservedRunningTime="2026-07-17 18:45:47.290893193 +0000 UTC m=+73.053323454"
Jul 17 18:45:47.506074 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.506031    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:47.511615 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.511543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d22bd50b-1802-473b-8dd7-1e5a96e7ff6b-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-pp97k\" (UID: \"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:47.795044 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.794972    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fhdlj\""
Jul 17 18:45:47.802771 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.802525    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"
Jul 17 18:45:47.980700 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.980625    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k"]
Jul 17 18:45:47.998663 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:47.998448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-wrcwt"
Jul 17 18:45:48.190234 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.190161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" event={"ID":"1f0543a3-7342-40cc-a87d-c261298366c4","Type":"ContainerStarted","Data":"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"}
Jul 17 18:45:48.193117 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.192806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9" event={"ID":"0a883a90-b8e3-46eb-a99d-eeded91debbd","Type":"ContainerStarted","Data":"abfbcf948aad59f9f8ebf7c1eb0f3785bc4423d223413f581085fb9ccb983f52"}
Jul 17 18:45:48.193117 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.192852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9" event={"ID":"0a883a90-b8e3-46eb-a99d-eeded91debbd","Type":"ContainerStarted","Data":"3427f77942b13ea0089f6da7afdad8cf50f0a76d39790052c794a1baa7223c7c"}
Jul 17 18:45:48.193117 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.192898    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:45:48.194606 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.194537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl" event={"ID":"42b75940-c085-4f8c-bce5-4d23c1f6a7b2","Type":"ContainerStarted","Data":"1fd824ef8e7ec760b061c6682eb39cd123b5dee183970651cd8b4a3368f321b4"}
Jul 17 18:45:48.197293 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.196780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm" event={"ID":"c4898006-61dc-42b4-a9d6-b3022d1adc4f","Type":"ContainerStarted","Data":"37e6c636f935f3789d44a42ef075fbeefe18e70d436a33cea2c65f01e903e640"}
Jul 17 18:45:48.197293 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.196818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm" event={"ID":"c4898006-61dc-42b4-a9d6-b3022d1adc4f","Type":"ContainerStarted","Data":"e6e6e2a8833c24e6319d80b4f1b76c6c1364300f90144a284c791745cd2c1916"}
Jul 17 18:45:48.199897 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.199868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k" event={"ID":"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b","Type":"ContainerStarted","Data":"551707c93920958221921865b4b384d89276101a6bfdca712b9d64b5a34830e0"}
Jul 17 18:45:48.202348 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.202244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j" event={"ID":"2001ea44-da85-4f05-b6a1-14552278aa80","Type":"ContainerStarted","Data":"96835f65ee8187be9654e6bdede81eb893438ef7e3a8712dfd8a05bd92d1af0e"}
Jul 17 18:45:48.224901 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.224845    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9" podStartSLOduration=2.224824158 podStartE2EDuration="2.224824158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:48.223925289 +0000 UTC m=+73.986355548" watchObservedRunningTime="2026-07-17 18:45:48.224824158 +0000 UTC m=+73.987254419"
Jul 17 18:45:48.247788 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.247719    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm" podStartSLOduration=60.247696167 podStartE2EDuration="1m0.247696167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:48.245875826 +0000 UTC m=+74.008306091" watchObservedRunningTime="2026-07-17 18:45:48.247696167 +0000 UTC m=+74.010126428"
Jul 17 18:45:48.826165 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.824867    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:48.828657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:48.828473    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:49.208252 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:49.208182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:49.209513 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:49.209492    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5c6cb4dc5b-f5hkm"
Jul 17 18:45:54.226460 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.226364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k" event={"ID":"d22bd50b-1802-473b-8dd7-1e5a96e7ff6b","Type":"ContainerStarted","Data":"9d10efd8a19f30eeaa88ef9381f161e20790d0202000689d1ab74bb590c701a5"}
Jul 17 18:45:54.228508 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.228470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j" event={"ID":"2001ea44-da85-4f05-b6a1-14552278aa80","Type":"ContainerStarted","Data":"1780b9431c6bca8273330393d8988f477d1b19c77bb0420ab0daac5dbbbe5725"}
Jul 17 18:45:54.228641 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.228541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j" event={"ID":"2001ea44-da85-4f05-b6a1-14552278aa80","Type":"ContainerStarted","Data":"984f5d9a6b64431af8a0167f8b6ba9e021da68add2e4ecaefdbb119345c71b23"}
Jul 17 18:45:54.230343 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.230296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qkzh" event={"ID":"87336d0b-4178-4b44-b999-a44ed8434de8","Type":"ContainerStarted","Data":"ac5a095c023e1cdf3ef61943178b9da01becfebc7f17f88c2b462a6820eadba2"}
Jul 17 18:45:54.232062 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.232031    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hph6v" event={"ID":"f8be0c3f-c344-4968-92da-6e028a539e6f","Type":"ContainerStarted","Data":"af5c95a8f9f7fcf242cff2429d552043a5d1404d7f97f51a978a3d1b68218c55"}
Jul 17 18:45:54.232158 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.232083    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hph6v"
Jul 17 18:45:54.232158 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.232107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hph6v" event={"ID":"f8be0c3f-c344-4968-92da-6e028a539e6f","Type":"ContainerStarted","Data":"07678d78e586f3aa963a8d6ed11b89c501876da58319d3b944b3a21bd61c6342"}
Jul 17 18:45:54.233628 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.233605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl" event={"ID":"42b75940-c085-4f8c-bce5-4d23c1f6a7b2","Type":"ContainerStarted","Data":"ba256262c5e4dff0299e22f7a579677a3a4dec07d1e112e5e70300741c04c5ab"}
Jul 17 18:45:54.245861 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.245815    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-pp97k" podStartSLOduration=33.675349299 podStartE2EDuration="39.245801077s" totalImagesPullingTime="5.570451778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:54.244647643 +0000 UTC m=+80.007077906" watchObservedRunningTime="2026-07-17 18:45:54.245801077 +0000 UTC m=+80.008231314"
Jul 17 18:45:54.269040 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.267380    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-sms5j" podStartSLOduration=60.026044479 podStartE2EDuration="1m6.267359287s" totalImagesPullingTime="6.241314808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:54.261638748 +0000 UTC m=+80.024069007" watchObservedRunningTime="2026-07-17 18:45:54.267359287 +0000 UTC m=+80.029789549"
Jul 17 18:45:54.282023 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.281975    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-b4pxl" podStartSLOduration=59.987197663 podStartE2EDuration="1m6.281959086s" totalImagesPullingTime="6.294761423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:54.280428557 +0000 UTC m=+80.042858846" watchObservedRunningTime="2026-07-17 18:45:54.281959086 +0000 UTC m=+80.044389345"
Jul 17 18:45:54.302402 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:54.302358    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hph6v" podStartSLOduration=36.765843846 podStartE2EDuration="43.302341866s" totalImagesPullingTime="6.53649802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:54.301169406 +0000 UTC m=+80.063599662" watchObservedRunningTime="2026-07-17 18:45:54.302341866 +0000 UTC m=+80.064772137"
Jul 17 18:45:58.252003 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:58.251969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qkzh" event={"ID":"87336d0b-4178-4b44-b999-a44ed8434de8","Type":"ContainerStarted","Data":"1a193133278926453834c14c9e43c7e7227ec6d36cc85a47e6745d567b74b4af"}
Jul 17 18:45:58.273761 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:45:58.273707    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6qkzh" podStartSLOduration=17.511080834 podStartE2EDuration="35.273691991s" totalImagesPullingTime="17.762611157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:45:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:58.271868903 +0000 UTC m=+84.034299164" watchObservedRunningTime="2026-07-17 18:45:58.273691991 +0000 UTC m=+84.036122250"
Jul 17 18:46:04.240019 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:04.239989    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hph6v"
Jul 17 18:46:06.446772 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:06.446737    2558 patch_prober.go:28] interesting pod/image-registry-84fb5db76b-f97n9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 18:46:06.447165 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:06.446789    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9" podUID="0a883a90-b8e3-46eb-a99d-eeded91debbd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 18:46:07.186683 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:07.186649    2558 patch_prober.go:28] interesting pod/image-registry-6c777bf7d5-pln4f container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 18:46:07.186829 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:07.186701    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 18:46:09.211811 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:09.211785    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84fb5db76b-f97n9"
Jul 17 18:46:12.155093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.155062    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sqfrw"
Jul 17 18:46:12.204790 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.204754    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" containerName="registry" containerID="cri-o://20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01" gracePeriod=30
Jul 17 18:46:12.446036 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.446016    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:46:12.510398 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510525 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6bz9\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510525 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510451    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510625 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510558    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510625 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510605    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510729 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510633    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510729 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510729 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token\") pod \"1f0543a3-7342-40cc-a87d-c261298366c4\" (UID: \"1f0543a3-7342-40cc-a87d-c261298366c4\") "
Jul 17 18:46:12.510870 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510837    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:46:12.510961 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.510941    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-trusted-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.511125 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.511037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:46:12.513225 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.513201    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:46:12.513375 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.513343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:46:12.513467 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.513441    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9" (OuterVolumeSpecName: "kube-api-access-w6bz9") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "kube-api-access-w6bz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:46:12.513549 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.513521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:46:12.513687 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.513664    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:46:12.518962 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.518941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1f0543a3-7342-40cc-a87d-c261298366c4" (UID: "1f0543a3-7342-40cc-a87d-c261298366c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:46:12.611886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611847    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1f0543a3-7342-40cc-a87d-c261298366c4-ca-trust-extracted\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.611886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611877    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-bound-sa-token\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.611886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611888    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-registry-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.611886 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611896    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bz9\" (UniqueName: \"kubernetes.io/projected/1f0543a3-7342-40cc-a87d-c261298366c4-kube-api-access-w6bz9\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.612142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611905    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-installation-pull-secrets\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.612142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611916    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1f0543a3-7342-40cc-a87d-c261298366c4-image-registry-private-configuration\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:12.612142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:12.611925    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1f0543a3-7342-40cc-a87d-c261298366c4-registry-certificates\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:46:13.306667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.306635    2558 generic.go:356] "Generic (PLEG): container finished" podID="1f0543a3-7342-40cc-a87d-c261298366c4" containerID="20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01" exitCode=0
Jul 17 18:46:13.307094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.306682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" event={"ID":"1f0543a3-7342-40cc-a87d-c261298366c4","Type":"ContainerDied","Data":"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"}
Jul 17 18:46:13.307094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.306689    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f"
Jul 17 18:46:13.307094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.306721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c777bf7d5-pln4f" event={"ID":"1f0543a3-7342-40cc-a87d-c261298366c4","Type":"ContainerDied","Data":"ac688228fc9bd044e060decef9018718e9c9bd232dac6dbe3068a6fe7c21a9cb"}
Jul 17 18:46:13.307094 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.306740    2558 scope.go:117] "RemoveContainer" containerID="20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"
Jul 17 18:46:13.314950 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.314931    2558 scope.go:117] "RemoveContainer" containerID="20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"
Jul 17 18:46:13.315231 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:46:13.315210    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01\": container with ID starting with 20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01 not found: ID does not exist" containerID="20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"
Jul 17 18:46:13.315300 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.315244    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01"} err="failed to get container status \"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01\": rpc error: code = NotFound desc = could not find container \"20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01\": container with ID starting with 20d160e8d943952393ef06c0d01be29d94670cee970234eac8ff87ef4c288a01 not found: ID does not exist"
Jul 17 18:46:13.336268 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.336236    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:46:13.344411 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:13.344388    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c777bf7d5-pln4f"]
Jul 17 18:46:14.771670 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:14.771635    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" path="/var/lib/kubelet/pods/1f0543a3-7342-40cc-a87d-c261298366c4/volumes"
Jul 17 18:46:17.749844 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.749805    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-txcns"]
Jul 17 18:46:17.750291 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.750163    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" containerName="registry"
Jul 17 18:46:17.750291 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.750178    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" containerName="registry"
Jul 17 18:46:17.750291 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.750254    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f0543a3-7342-40cc-a87d-c261298366c4" containerName="registry"
Jul 17 18:46:17.770725 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.770701    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.773958 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.773937    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p9fbf\""
Jul 17 18:46:17.774733 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.774709    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 18:46:17.774826 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.774749    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 18:46:17.774826 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.774717    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 18:46:17.774916 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.774717    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 18:46:17.850919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.850888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-sys\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.850928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-root\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.850969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-metrics-client-ca\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.850994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-wtmp\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.851015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851093 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.851055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-textfile\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.851101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgr6l\" (UniqueName: \"kubernetes.io/projected/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-kube-api-access-dgr6l\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.851121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-tls\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.851254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.851138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951659 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-metrics-client-ca\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951659 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-wtmp\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951703    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-textfile\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgr6l\" (UniqueName: \"kubernetes.io/projected/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-kube-api-access-dgr6l\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-tls\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-sys\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-wtmp\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.951884 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-root\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.952273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-sys\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.952273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.951952    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-root\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.952273 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.952125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-textfile\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.952424 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.952352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-metrics-client-ca\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.952424 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.952345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-accelerators-collector-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.954384 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.954341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.954494 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.954432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-node-exporter-tls\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:17.960797 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:17.960776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgr6l\" (UniqueName: \"kubernetes.io/projected/ba5cf590-e4bf-45a3-a9a0-45741a7301c8-kube-api-access-dgr6l\") pod \"node-exporter-txcns\" (UID: \"ba5cf590-e4bf-45a3-a9a0-45741a7301c8\") " pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:18.079681 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:18.079653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-txcns"
Jul 17 18:46:18.091902 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:46:18.091871    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5cf590_e4bf_45a3_a9a0_45741a7301c8.slice/crio-3d74acedaa384e06ea3be109629ee699bbcbbfeb965b4074ff4262459080a1f1 WatchSource:0}: Error finding container 3d74acedaa384e06ea3be109629ee699bbcbbfeb965b4074ff4262459080a1f1: Status 404 returned error can't find the container with id 3d74acedaa384e06ea3be109629ee699bbcbbfeb965b4074ff4262459080a1f1
Jul 17 18:46:18.327358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:18.327300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txcns" event={"ID":"ba5cf590-e4bf-45a3-a9a0-45741a7301c8","Type":"ContainerStarted","Data":"3d74acedaa384e06ea3be109629ee699bbcbbfeb965b4074ff4262459080a1f1"}
Jul 17 18:46:20.336981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:20.336910    2558 generic.go:356] "Generic (PLEG): container finished" podID="ba5cf590-e4bf-45a3-a9a0-45741a7301c8" containerID="78937c4fceabf7c6cf3d372d5e9d806264c1bcdbe201cf269210ff4ee9fb9685" exitCode=0
Jul 17 18:46:20.337471 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:20.336996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txcns" event={"ID":"ba5cf590-e4bf-45a3-a9a0-45741a7301c8","Type":"ContainerDied","Data":"78937c4fceabf7c6cf3d372d5e9d806264c1bcdbe201cf269210ff4ee9fb9685"}
Jul 17 18:46:21.343216 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:21.343177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txcns" event={"ID":"ba5cf590-e4bf-45a3-a9a0-45741a7301c8","Type":"ContainerStarted","Data":"58a140b72119b367a88ab54a5ddca787d31cea083b0858f4fbd5e7bce288ce34"}
Jul 17 18:46:21.343607 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:21.343236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-txcns" event={"ID":"ba5cf590-e4bf-45a3-a9a0-45741a7301c8","Type":"ContainerStarted","Data":"1e331bb53a41660dfa6c456bdd9f84f4fbd3b90b46a8d6ce5b773eb375e70342"}
Jul 17 18:46:21.392627 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:21.392576    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-txcns" podStartSLOduration=3.091639004 podStartE2EDuration="4.392559145s" totalImagesPullingTime="1.227314824s" totalInitContainerRuntime="73.605317ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:21.392391309 +0000 UTC m=+107.154821580" watchObservedRunningTime="2026-07-17 18:46:21.392559145 +0000 UTC m=+107.154989406"
Jul 17 18:46:24.103777 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.103735    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:46:24.118395 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.118374    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.123284 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.123260    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 18:46:24.123441 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.123269    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 18:46:24.123561 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.123548    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 18:46:24.123605 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.123560    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 18:46:24.126603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.126577    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 18:46:24.126762 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.126746    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 18:46:24.126841 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.126796    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 18:46:24.127100 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.127086    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 18:46:24.127908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.127885    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 18:46:24.128009 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.127915    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvqhp\""
Jul 17 18:46:24.128009 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.127962    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-499l2gfqbuc9j\""
Jul 17 18:46:24.128115 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.128087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 18:46:24.129491 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.129472    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 18:46:24.130751 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.130721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 18:46:24.131373 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.131108    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:46:24.131373 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.131109    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 18:46:24.132876 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.132854    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 18:46:24.133338 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.133300    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 18:46:24.201992 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.201960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.201992 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.201996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp42p\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202263    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202430 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202380    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.202612 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.202603    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303334 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303524 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303508    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp42p\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.303908 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.303865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.304719 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.304177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.306870 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.306764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.306870 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.306764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.306870 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.306839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.307115 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.306963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.307115 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.307062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.307213 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.307196    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.307263 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.307234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.307439 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.307415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.308052 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.308029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.312142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.312121    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.315167 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.315147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.316283 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.316152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.318096 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.318073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.318181 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.318146    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.318222 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.318186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.318285 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.318268    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.318285 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.318280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.322784 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.322762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp42p\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.323927 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.323904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config\") pod \"prometheus-k8s-0\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.427655 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.427578    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:24.600750 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:24.600718    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:46:24.603391 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:46:24.603364    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82450902_dc13_4d3f_8c5f_3004bbd92bd6.slice/crio-63e9e99d25c3db16e0c94f51efbc6018e497a8fbd40aafe041cd998f7ef094b1 WatchSource:0}: Error finding container 63e9e99d25c3db16e0c94f51efbc6018e497a8fbd40aafe041cd998f7ef094b1: Status 404 returned error can't find the container with id 63e9e99d25c3db16e0c94f51efbc6018e497a8fbd40aafe041cd998f7ef094b1
Jul 17 18:46:25.359706 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:25.359667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"63e9e99d25c3db16e0c94f51efbc6018e497a8fbd40aafe041cd998f7ef094b1"}
Jul 17 18:46:26.365795 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:26.365755    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e" exitCode=0
Jul 17 18:46:26.366274 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:26.365835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"}
Jul 17 18:46:30.385244 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:30.385158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"}
Jul 17 18:46:30.385649 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:30.385276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"}
Jul 17 18:46:32.396658 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:32.396624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"}
Jul 17 18:46:32.397046 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:32.396703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"}
Jul 17 18:46:32.397046 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:32.396742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"}
Jul 17 18:46:32.397046 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:32.396797    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerStarted","Data":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"}
Jul 17 18:46:32.439513 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:32.439461    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.2780238179999999 podStartE2EDuration="8.43943672s" totalImagesPullingTime="7.154644587s" totalInitContainerRuntime="6.768315ms" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:32.437544069 +0000 UTC m=+118.199974330" watchObservedRunningTime="2026-07-17 18:46:32.43943672 +0000 UTC m=+118.201866982"
Jul 17 18:46:33.402363 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.402266    2558 generic.go:356] "Generic (PLEG): container finished" podID="2bae47ed-4f68-4b6f-b482-6ed4210d1118" containerID="6da3bc2dfc4f77b95561550d00f294c4dd2aecae2a4957cc669c33329192166a" exitCode=0
Jul 17 18:46:33.402363 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.402344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2" event={"ID":"2bae47ed-4f68-4b6f-b482-6ed4210d1118","Type":"ContainerDied","Data":"6da3bc2dfc4f77b95561550d00f294c4dd2aecae2a4957cc669c33329192166a"}
Jul 17 18:46:33.402851 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.402690    2558 scope.go:117] "RemoveContainer" containerID="6da3bc2dfc4f77b95561550d00f294c4dd2aecae2a4957cc669c33329192166a"
Jul 17 18:46:33.404025 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.404001    2558 generic.go:356] "Generic (PLEG): container finished" podID="954c5632-fd22-4e9b-9536-b8f6dd21b5c0" containerID="eba0698d9958d75e36eff669fbc1540cdd0e8bba355cd38975c5b86159deac58" exitCode=0
Jul 17 18:46:33.404084 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.404066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cm2gp" event={"ID":"954c5632-fd22-4e9b-9536-b8f6dd21b5c0","Type":"ContainerDied","Data":"eba0698d9958d75e36eff669fbc1540cdd0e8bba355cd38975c5b86159deac58"}
Jul 17 18:46:33.404375 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:33.404360    2558 scope.go:117] "RemoveContainer" containerID="eba0698d9958d75e36eff669fbc1540cdd0e8bba355cd38975c5b86159deac58"
Jul 17 18:46:34.410525 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:34.410486    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-m87h2" event={"ID":"2bae47ed-4f68-4b6f-b482-6ed4210d1118","Type":"ContainerStarted","Data":"1dc05550c249fb41fd218a28eb5478f6ea2f76e20b1435e3bdd3baa1c8d3df2a"}
Jul 17 18:46:34.412264 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:34.412244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-cm2gp" event={"ID":"954c5632-fd22-4e9b-9536-b8f6dd21b5c0","Type":"ContainerStarted","Data":"78ceabf609cc8b412a6cf9cf99f16c1fe0c7baae49ae2037a007dbe9b7b73518"}
Jul 17 18:46:34.428254 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:34.428224    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:46:42.444658 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:42.444625    2558 generic.go:356] "Generic (PLEG): container finished" podID="d6e23da7-45bb-4993-ba2b-e82954181fbb" containerID="a2b6098a1b18b56fd485eef65e943b60311a267c2156b7b81ed69b35465c17eb" exitCode=0
Jul 17 18:46:42.445074 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:42.444705    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l" event={"ID":"d6e23da7-45bb-4993-ba2b-e82954181fbb","Type":"ContainerDied","Data":"a2b6098a1b18b56fd485eef65e943b60311a267c2156b7b81ed69b35465c17eb"}
Jul 17 18:46:42.445074 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:42.445066    2558 scope.go:117] "RemoveContainer" containerID="a2b6098a1b18b56fd485eef65e943b60311a267c2156b7b81ed69b35465c17eb"
Jul 17 18:46:43.450333 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:46:43.450280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-2g79l" event={"ID":"d6e23da7-45bb-4993-ba2b-e82954181fbb","Type":"ContainerStarted","Data":"927005b3f339af8a8146b442d5490da9cf1be15579c742a621de53b414ce37e0"}
Jul 17 18:47:24.428819 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:24.428771    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:24.445098 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:24.445069    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:24.621103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:24.621077    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:42.716490 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716398    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716714    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="prometheus" containerID="cri-o://5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414" gracePeriod=600
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716753    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy" containerID="cri-o://396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb" gracePeriod=600
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716771    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="thanos-sidecar" containerID="cri-o://e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91" gracePeriod=600
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716789    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-web" containerID="cri-o://9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c" gracePeriod=600
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716806    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="config-reloader" containerID="cri-o://68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf" gracePeriod=600
Jul 17 18:47:42.716947 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.716887    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-thanos" containerID="cri-o://54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda" gracePeriod=600
Jul 17 18:47:42.741057 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:42.741030    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82450902_dc13_4d3f_8c5f_3004bbd92bd6.slice/crio-9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 18:47:42.962277 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.962251    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:42.996474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996399    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.996474 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996434    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.996713 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996488    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.996713 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996619    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.996713 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996662    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.996713 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996691    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.997005 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996721    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.997887 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.997147    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:42.998098 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.998051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:47:42.999441 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.996763    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999551 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999480    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:42.999551 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999491    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999671 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999568    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp42p\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999671 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999671 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999665    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999691    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999771    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:42.999830 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999798    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:43.000134 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:43.000134 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:43.000134 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:42.999885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0\") pod \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\" (UID: \"82450902-dc13-4d3f-8c5f-3004bbd92bd6\") "
Jul 17 18:47:43.000290 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000136    2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-kube-rbac-proxy\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.000290 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000158    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-db\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.000290 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000173    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.000290 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000186    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-metrics-client-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.000290 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000256    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:43.001155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:43.001155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000691    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:43.001155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000756    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.001155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.000918    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:43.001580 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.001554    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 18:47:43.002068 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.002042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config" (OuterVolumeSpecName: "config") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.002953 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.002929    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out" (OuterVolumeSpecName: "config-out") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:47:43.003602 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.003566    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.003694 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.003672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p" (OuterVolumeSpecName: "kube-api-access-xp42p") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "kube-api-access-xp42p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:47:43.004567 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.004533    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 18:47:43.004750 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.004719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.004925 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.004901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.005258 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.005231    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.006049 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.006021    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.014945 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.014920    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config" (OuterVolumeSpecName: "web-config") pod "82450902-dc13-4d3f-8c5f-3004bbd92bd6" (UID: "82450902-dc13-4d3f-8c5f-3004bbd92bd6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 18:47:43.100539 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100506    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100539 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100533    2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-grpc-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100539 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100543    2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-metrics-client-certs\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100553    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100563    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100572    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-tls-assets\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100580    2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100588    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100597    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100607    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100616    2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100624    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/82450902-dc13-4d3f-8c5f-3004bbd92bd6-config-out\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100632    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100639    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/82450902-dc13-4d3f-8c5f-3004bbd92bd6-web-config\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100647    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp42p\" (UniqueName: \"kubernetes.io/projected/82450902-dc13-4d3f-8c5f-3004bbd92bd6-kube-api-access-xp42p\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.100747 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.100654    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/82450902-dc13-4d3f-8c5f-3004bbd92bd6-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:47:43.677898 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677864    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda" exitCode=0
Jul 17 18:47:43.677898 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677896    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb" exitCode=0
Jul 17 18:47:43.677898 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677905    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c" exitCode=0
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677913    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91" exitCode=0
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677924    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf" exitCode=0
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677931    2558 generic.go:356] "Generic (PLEG): container finished" podID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414" exitCode=0
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677935    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"}
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.677962    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"}
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"}
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678093    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"}
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"}
Jul 17 18:47:43.678202 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678192    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"}
Jul 17 18:47:43.678708 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.678232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"82450902-dc13-4d3f-8c5f-3004bbd92bd6","Type":"ContainerDied","Data":"63e9e99d25c3db16e0c94f51efbc6018e497a8fbd40aafe041cd998f7ef094b1"}
Jul 17 18:47:43.686667 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.686647    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.694017 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.694004    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.700888 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.700874    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.703068 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.703047    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:43.708171 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.708100    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.710227 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.710209    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:43.716036 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.715941    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.723753 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.723578    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.730194 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.730171    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.730474 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.730456    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.730536 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.730483    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.730536 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.730501    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.730777 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.730752    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.730825 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.730789    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.730825 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.730813    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.731098 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.731081    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.731164 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731116    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.731164 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731138    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.731407 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.731389    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.731463 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731410    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.731463 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731424    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.731662 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.731642    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.731706 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731671    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.731706 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731692    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.731924 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.731904    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.731962 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731928    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.731962 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.731943    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.732146 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:47:43.732127    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.732188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732151    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.732188 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732170    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.732414 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732396    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.732470 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732415    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.732641 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732618    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.732641 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732641    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.733048 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732834    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.733048 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.732857    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.733190 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733073    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.733190 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733087    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.733384 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733276    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.733384 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733297    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.733607 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733530    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.733607 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733550    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.733897 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733792    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.733897 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733805    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.734095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.733984    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.734095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734001    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.734292 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734224    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.734292 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734238    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.734565 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734455    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.734565 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734477    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.734868 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734774    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.734868 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.734790    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.735079 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735053    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.735185 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735081    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.735381 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735355    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.735454 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735384    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.735687 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735663    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.735770 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.735686    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.736056 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736033    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.736142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736058    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.736358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736309    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.736358 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736358    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.736705 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736685    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.736809 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.736707    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.737036 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737017    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.737036 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737037    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.737344 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737306    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.737344 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737344    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.737639 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737603    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.737712 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737639    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.737888 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737866    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.737888 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.737888    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.738161 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738141    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.738236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738165    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.738462 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738443    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.738540 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738463    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.738707 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738685    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.738707 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738705    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.738957 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738933    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.739045 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.738959    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.739210 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739189    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.739210 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739207    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.739472 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739439    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.739472 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739461    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.739702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739679    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.739702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739702    2558 scope.go:117] "RemoveContainer" containerID="54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"
Jul 17 18:47:43.739984 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739967    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda"} err="failed to get container status \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": rpc error: code = NotFound desc = could not find container \"54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda\": container with ID starting with 54abc0bbd81cca2a35f1c0b15f66254672a3629a74c4a134e3ca7c524f255fda not found: ID does not exist"
Jul 17 18:47:43.740045 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.739985    2558 scope.go:117] "RemoveContainer" containerID="396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"
Jul 17 18:47:43.740222 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740203    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb"} err="failed to get container status \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": rpc error: code = NotFound desc = could not find container \"396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb\": container with ID starting with 396b58348b2127b8e179840755e5dde9c7e9dbd8cda933c04761be599d914bfb not found: ID does not exist"
Jul 17 18:47:43.740287 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740222    2558 scope.go:117] "RemoveContainer" containerID="9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"
Jul 17 18:47:43.740452 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740435    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c"} err="failed to get container status \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": rpc error: code = NotFound desc = could not find container \"9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c\": container with ID starting with 9b7b4c8948616a3e3a782640f6d17d550c2304f1a2c42fe2340167d0d065d40c not found: ID does not exist"
Jul 17 18:47:43.740504 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740455    2558 scope.go:117] "RemoveContainer" containerID="e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"
Jul 17 18:47:43.740702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740682    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91"} err="failed to get container status \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": rpc error: code = NotFound desc = could not find container \"e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91\": container with ID starting with e191072dee8f1361b3c970bc23bdd6b21d0eddd9e04c31b935525fa7db4a2b91 not found: ID does not exist"
Jul 17 18:47:43.740702 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740701    2558 scope.go:117] "RemoveContainer" containerID="68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"
Jul 17 18:47:43.740958 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740934    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf"} err="failed to get container status \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": rpc error: code = NotFound desc = could not find container \"68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf\": container with ID starting with 68de173cc2fb7931ff3178892003781296d23da42ab7393622e0289d4e0c59bf not found: ID does not exist"
Jul 17 18:47:43.741022 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.740958    2558 scope.go:117] "RemoveContainer" containerID="5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"
Jul 17 18:47:43.741195 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.741177    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414"} err="failed to get container status \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": rpc error: code = NotFound desc = could not find container \"5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414\": container with ID starting with 5e327f504a8f28a76e90f8660816e9eada550c2e96254bc4e57fadaa2d54f414 not found: ID does not exist"
Jul 17 18:47:43.741297 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.741196    2558 scope.go:117] "RemoveContainer" containerID="9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"
Jul 17 18:47:43.741490 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.741467    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e"} err="failed to get container status \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": rpc error: code = NotFound desc = could not find container \"9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e\": container with ID starting with 9821f9d4828b6fa37ec3188a354e42e8684f8043dca1a35368dc0b92de1a064e not found: ID does not exist"
Jul 17 18:47:43.749814 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.749794    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:43.750059 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750048    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-web"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750062    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-web"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750070    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-thanos"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750075    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-thanos"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750081    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="config-reloader"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750087    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="config-reloader"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750093    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750099    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy"
Jul 17 18:47:43.750103 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750105    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="prometheus"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750110    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="prometheus"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750116    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="thanos-sidecar"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750121    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="thanos-sidecar"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750135    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="init-config-reloader"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750141    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="init-config-reloader"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750185    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="prometheus"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750192    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750198    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="config-reloader"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750203    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="thanos-sidecar"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750208    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-web"
Jul 17 18:47:43.750347 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.750213    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" containerName="kube-rbac-proxy-thanos"
Jul 17 18:47:43.755402 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.755383    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.758308 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.758288    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 17 18:47:43.758476 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.758461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 17 18:47:43.762505 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.762490    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 17 18:47:43.763302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763265    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-499l2gfqbuc9j\""
Jul 17 18:47:43.763558 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763542    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 17 18:47:43.763735 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763710    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 17 18:47:43.763797 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763749    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 17 18:47:43.763911 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763896    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 17 18:47:43.763970 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.763945    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 17 18:47:43.764221 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.764206    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 17 18:47:43.764514 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.764502    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 17 18:47:43.765198 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.765186    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 17 18:47:43.766190 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.766176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 17 18:47:43.766531 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.766514    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 17 18:47:43.766621 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.766608    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvqhp\""
Jul 17 18:47:43.766678 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.766633    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 17 18:47:43.770682 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.770665    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 17 18:47:43.786972 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.786949    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:43.804558 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804534    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804678 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804564    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804744 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804731    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804802 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804798    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804918 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.804972 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805021 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.804972    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805091 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805146 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805146 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-config-out\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805248 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805248 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805224    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-web-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805357 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805357 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805427 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t94\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-kube-api-access-k4t94\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805427 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805390    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805427 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.805511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.805457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.905981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.905940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.905981 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.905976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.905998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906033    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-config-out\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-web-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t94\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-kube-api-access-k4t94\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906217 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.906734 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.907095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.907095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.907095 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.906871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.907594 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.907308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.907740 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.907713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.908225 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.908194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.908482 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.908456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.910932 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.910650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.910932 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.910771    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-web-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.910932 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.910886    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.911210 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.911766 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1790c087-12de-43a8-9bec-c6ecca1e8054-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.911766 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.911929 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.911929 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.912041 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.911996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1790c087-12de-43a8-9bec-c6ecca1e8054-config-out\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.912444 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.912423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.912528 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.912428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.913019 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.913001    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1790c087-12de-43a8-9bec-c6ecca1e8054-config\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:43.915842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:43.915819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t94\" (UniqueName: \"kubernetes.io/projected/1790c087-12de-43a8-9bec-c6ecca1e8054-kube-api-access-k4t94\") pod \"prometheus-k8s-0\" (UID: \"1790c087-12de-43a8-9bec-c6ecca1e8054\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:44.064035 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.064003    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:47:44.209513 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.209479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 17 18:47:44.213173 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:47:44.213145    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1790c087_12de_43a8_9bec_c6ecca1e8054.slice/crio-f4a7bcf9d736e18832c4dec4e9bd040c9945a6fb33a53f44e3d769a1a8c28d1a WatchSource:0}: Error finding container f4a7bcf9d736e18832c4dec4e9bd040c9945a6fb33a53f44e3d769a1a8c28d1a: Status 404 returned error can't find the container with id f4a7bcf9d736e18832c4dec4e9bd040c9945a6fb33a53f44e3d769a1a8c28d1a
Jul 17 18:47:44.687189 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.687152    2558 generic.go:356] "Generic (PLEG): container finished" podID="1790c087-12de-43a8-9bec-c6ecca1e8054" containerID="4801d002a0ea54f5cc1dfdbd384a34d11fd722db40b14740710bd4a51116d72e" exitCode=0
Jul 17 18:47:44.687400 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.687243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerDied","Data":"4801d002a0ea54f5cc1dfdbd384a34d11fd722db40b14740710bd4a51116d72e"}
Jul 17 18:47:44.687400 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.687369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"f4a7bcf9d736e18832c4dec4e9bd040c9945a6fb33a53f44e3d769a1a8c28d1a"}
Jul 17 18:47:44.773486 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:44.773459    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82450902-dc13-4d3f-8c5f-3004bbd92bd6" path="/var/lib/kubelet/pods/82450902-dc13-4d3f-8c5f-3004bbd92bd6/volumes"
Jul 17 18:47:45.696299 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"237119b85a1cbcf2b967ac9f02f36a0022dc7f64a50a98a5e8e01462f290ad77"}
Jul 17 18:47:45.696544 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696349    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"cb4a9049fe4c39a54ddf4114e57626641f3f93c3c3a109c4b512b92ab33cc546"}
Jul 17 18:47:45.696544 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"fb8be96eb440e193b95f429d15d992ab53d3923f910cc363993b470daa316cb8"}
Jul 17 18:47:45.696544 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"0e1e25a215126d6a12eb3134bc8d4bb48531a394e39a077bd5e2f978419ba40b"}
Jul 17 18:47:45.696544 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"e1142852b0150cba5097138b280c34e6abc586d343f251be8609eb2427ff73a8"}
Jul 17 18:47:45.696544 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.696454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1790c087-12de-43a8-9bec-c6ecca1e8054","Type":"ContainerStarted","Data":"ee434f415560b5da6535f6ef67845b553ea6144bdee1e65456a306da8d83bfb9"}
Jul 17 18:47:45.729850 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:45.729789    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.70924906 podStartE2EDuration="2.729770865s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.521805ms" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:45.729057913 +0000 UTC m=+191.491488173" watchObservedRunningTime="2026-07-17 18:47:45.729770865 +0000 UTC m=+191.492201126"
Jul 17 18:47:49.064359 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:47:49.064291    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:48:44.064314 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:48:44.064276    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:48:44.080517 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:48:44.080493    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:48:44.946741 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:48:44.946714    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 17 18:49:34.661453 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:49:34.661429    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:49:34.667677 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:49:34.667653    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:49:34.672894 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:49:34.672874    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 18:51:08.844951 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.844919    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"]
Jul 17 18:51:08.848464 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.848447    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:08.852055 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.852031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 18:51:08.852260 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.852243    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 18:51:08.852644 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.852628    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 18:51:08.852745 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.852651    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 18:51:08.852862 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.852848    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mb6px\""
Jul 17 18:51:08.871576 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.871548    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"]
Jul 17 18:51:08.945575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.945539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:08.945752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.945582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-manifests\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:08.945752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.945600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:08.945752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.945664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2bz\" (UniqueName: \"kubernetes.io/projected/015b05c6-f57a-48e5-be54-d87ecde170f4-kube-api-access-8v2bz\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:08.945752 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:08.945725    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-tmp\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.046805 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.046773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.046805 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.046812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-manifests\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.047063 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.046831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.047063 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.046973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2bz\" (UniqueName: \"kubernetes.io/projected/015b05c6-f57a-48e5-be54-d87ecde170f4-kube-api-access-8v2bz\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.047063 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.047020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-tmp\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.047276 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.047256    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-manifests\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.049310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.049288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/015b05c6-f57a-48e5-be54-d87ecde170f4-tmp\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.049636 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.049615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.049715 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.049701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/015b05c6-f57a-48e5-be54-d87ecde170f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.064727 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.064702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2bz\" (UniqueName: \"kubernetes.io/projected/015b05c6-f57a-48e5-be54-d87ecde170f4-kube-api-access-8v2bz\") pod \"opendatahub-operator-controller-manager-68969b7c46-jzkbx\" (UID: \"015b05c6-f57a-48e5-be54-d87ecde170f4\") " pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.158869 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.158780    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:09.301503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.301470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"]
Jul 17 18:51:09.305304 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:51:09.305277    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015b05c6_f57a_48e5_be54_d87ecde170f4.slice/crio-7d821e5ed17d320b0cd92795a5fbfecdbd3ab8be953e697985fb9c8a2000f526 WatchSource:0}: Error finding container 7d821e5ed17d320b0cd92795a5fbfecdbd3ab8be953e697985fb9c8a2000f526: Status 404 returned error can't find the container with id 7d821e5ed17d320b0cd92795a5fbfecdbd3ab8be953e697985fb9c8a2000f526
Jul 17 18:51:09.307011 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.306996    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 18:51:09.482749 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:09.482657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx" event={"ID":"015b05c6-f57a-48e5-be54-d87ecde170f4","Type":"ContainerStarted","Data":"7d821e5ed17d320b0cd92795a5fbfecdbd3ab8be953e697985fb9c8a2000f526"}
Jul 17 18:51:12.500490 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:12.500405    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx" event={"ID":"015b05c6-f57a-48e5-be54-d87ecde170f4","Type":"ContainerStarted","Data":"19f81d1c1f63267bece9eb4020b6bad1786d1d9863872b88bc6c5e16633ad827"}
Jul 17 18:51:13.505912 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:13.505880    2558 generic.go:356] "Generic (PLEG): container finished" podID="015b05c6-f57a-48e5-be54-d87ecde170f4" containerID="19f81d1c1f63267bece9eb4020b6bad1786d1d9863872b88bc6c5e16633ad827" exitCode=0
Jul 17 18:51:13.506302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:13.505964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx" event={"ID":"015b05c6-f57a-48e5-be54-d87ecde170f4","Type":"ContainerDied","Data":"19f81d1c1f63267bece9eb4020b6bad1786d1d9863872b88bc6c5e16633ad827"}
Jul 17 18:51:14.513424 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:14.513381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx" event={"ID":"015b05c6-f57a-48e5-be54-d87ecde170f4","Type":"ContainerStarted","Data":"1ab4ac5dd90e25367db59f249dcac01517c6f531453f295031aeaf45b1750cea"}
Jul 17 18:51:14.513852 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:14.513561    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:14.560301 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:14.560249    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx" podStartSLOduration=2.200534667 podStartE2EDuration="6.560227032s" totalImagesPullingTime="3.267544847s" totalInitContainerRuntime="1.092147518s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:14.557524891 +0000 UTC m=+400.319955150" watchObservedRunningTime="2026-07-17 18:51:14.560227032 +0000 UTC m=+400.322657293"
Jul 17 18:51:25.520339 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:25.520287    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68969b7c46-jzkbx"
Jul 17 18:51:36.884910 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.884874    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"]
Jul 17 18:51:36.889575 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.889556    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.895517 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.895496    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 18:51:36.895951 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.895816    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 18:51:36.895951 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.895817    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 18:51:36.897763 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.897740    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 18:51:36.898012 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.897990    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8rg78\""
Jul 17 18:51:36.924378 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.924347    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"]
Jul 17 18:51:36.993931 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.993902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994096 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.993937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfxrq\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-kube-api-access-wfxrq\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994096 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.994005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994096 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.994047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994258 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.994148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994258 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.994229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad983e-b65c-4529-94d1-23890da216fb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:36.994385 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:36.994272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095621 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095587    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad983e-b65c-4529-94d1-23890da216fb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095621 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfxrq\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-kube-api-access-wfxrq\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.095879 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.096112 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.095934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.096470 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.096446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.098207 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.098180    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.098305 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.098268    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.098481 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.098467    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fad983e-b65c-4529-94d1-23890da216fb-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.098640 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.098622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4fad983e-b65c-4529-94d1-23890da216fb-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.106000 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.105977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfxrq\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-kube-api-access-wfxrq\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.106670 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.106650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fad983e-b65c-4529-94d1-23890da216fb-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5jsdm\" (UID: \"4fad983e-b65c-4529-94d1-23890da216fb\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.198857 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.198770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:37.229434 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.227347    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"]
Jul 17 18:51:37.238079 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.238040    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.243091 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.243057    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 18:51:37.244367 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.244341    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 18:51:37.244527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.244395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 18:51:37.244527 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.244353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 18:51:37.244672 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.244529    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cw8fk\""
Jul 17 18:51:37.247225 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.247209    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 18:51:37.252642 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.252617    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"]
Jul 17 18:51:37.359362 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:51:37.359309    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fad983e_b65c_4529_94d1_23890da216fb.slice/crio-d4397c5e54aa3a8f06bda538447557ca3a1ef3da2eac165f2b42e3ab7d0cbfc3 WatchSource:0}: Error finding container d4397c5e54aa3a8f06bda538447557ca3a1ef3da2eac165f2b42e3ab7d0cbfc3: Status 404 returned error can't find the container with id d4397c5e54aa3a8f06bda538447557ca3a1ef3da2eac165f2b42e3ab7d0cbfc3
Jul 17 18:51:37.360634 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.360606    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"]
Jul 17 18:51:37.398399 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.398366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-metrics-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.398399 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.398408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.398616 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.398426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvdl\" (UniqueName: \"kubernetes.io/projected/46b4f2e4-ced5-48e7-a52b-a622514933ce-kube-api-access-rqvdl\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.398616 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.398544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46b4f2e4-ced5-48e7-a52b-a622514933ce-manager-config\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.500034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.499942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-metrics-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.500034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.499979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.500034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.499995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvdl\" (UniqueName: \"kubernetes.io/projected/46b4f2e4-ced5-48e7-a52b-a622514933ce-kube-api-access-rqvdl\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.500034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.500034    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46b4f2e4-ced5-48e7-a52b-a622514933ce-manager-config\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.500692 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.500669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/46b4f2e4-ced5-48e7-a52b-a622514933ce-manager-config\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.502603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.502580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-metrics-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.502603 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.502597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46b4f2e4-ced5-48e7-a52b-a622514933ce-cert\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.538278 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.538252    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvdl\" (UniqueName: \"kubernetes.io/projected/46b4f2e4-ced5-48e7-a52b-a622514933ce-kube-api-access-rqvdl\") pod \"lws-controller-manager-76fdf749bb-vf2bt\" (UID: \"46b4f2e4-ced5-48e7-a52b-a622514933ce\") " pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.549151 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.549131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:37.612339 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.612284    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm" event={"ID":"4fad983e-b65c-4529-94d1-23890da216fb","Type":"ContainerStarted","Data":"d4397c5e54aa3a8f06bda538447557ca3a1ef3da2eac165f2b42e3ab7d0cbfc3"}
Jul 17 18:51:37.717519 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:37.717496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"]
Jul 17 18:51:37.719845 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:51:37.719814    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b4f2e4_ced5_48e7_a52b_a622514933ce.slice/crio-824d95d04a5a53887fdc3269972b4696cc34e0752e268b94f7488dd452a1244a WatchSource:0}: Error finding container 824d95d04a5a53887fdc3269972b4696cc34e0752e268b94f7488dd452a1244a: Status 404 returned error can't find the container with id 824d95d04a5a53887fdc3269972b4696cc34e0752e268b94f7488dd452a1244a
Jul 17 18:51:38.619654 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:38.619583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt" event={"ID":"46b4f2e4-ced5-48e7-a52b-a622514933ce","Type":"ContainerStarted","Data":"824d95d04a5a53887fdc3269972b4696cc34e0752e268b94f7488dd452a1244a"}
Jul 17 18:51:40.630510 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:40.630407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt" event={"ID":"46b4f2e4-ced5-48e7-a52b-a622514933ce","Type":"ContainerStarted","Data":"fc3a304a7d01fa6b3695d36ad77675c20a0297abd51ba83e6f0bacabbae2d279"}
Jul 17 18:51:40.630922 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:40.630512    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:51:40.675978 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:40.675927    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt" podStartSLOduration=1.042672957 podStartE2EDuration="3.675910722s" totalImagesPullingTime="2.633237765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:40.673305343 +0000 UTC m=+426.435735602" watchObservedRunningTime="2026-07-17 18:51:40.675910722 +0000 UTC m=+426.438340981"
Jul 17 18:51:44.064684 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:44.064646    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 18:51:44.064978 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:44.064716    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 17 18:51:44.657142 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:44.657107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm" event={"ID":"4fad983e-b65c-4529-94d1-23890da216fb","Type":"ContainerStarted","Data":"c56380546411489ba0c4df52f7e6a0ef0a90bde831ba7bdb874b08afe18000b2"}
Jul 17 18:51:44.657369 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:44.657213    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:44.696618 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:44.696565    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm" podStartSLOduration=1.9934369969999999 podStartE2EDuration="8.696547448s" totalImagesPullingTime="6.703110451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:51:44.687938899 +0000 UTC m=+430.450369160" watchObservedRunningTime="2026-07-17 18:51:44.696547448 +0000 UTC m=+430.458977708"
Jul 17 18:51:45.662919 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:45.662890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5jsdm"
Jul 17 18:51:51.638546 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:51:51.638508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76fdf749bb-vf2bt"
Jul 17 18:53:28.088224 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.088139    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"]
Jul 17 18:53:28.091566 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.091538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.095031 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.095007    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 18:53:28.095155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.095050    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 18:53:28.095155 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.095124    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sfrlg\""
Jul 17 18:53:28.108654 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.108625    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"]
Jul 17 18:53:28.149893 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.149860    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7hdd\" (UniqueName: \"kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.150084 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.149923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.238450 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.238416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"]
Jul 17 18:53:28.238828 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:28.238801    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-q7hdd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv" podUID="bfbbeb1d-99a0-4d99-be55-0750cec53033"
Jul 17 18:53:28.250995 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.250971    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"]
Jul 17 18:53:28.251203 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.251179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hdd\" (UniqueName: \"kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.251300 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.251236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.251644 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.251628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.254115 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:28.254097    2558 projected.go:196] Error preparing data for projected volume kube-api-access-q7hdd for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-140.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object
Jul 17 18:53:28.254213 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:28.254167    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd podName:bfbbeb1d-99a0-4d99-be55-0750cec53033 nodeName:}" failed. No retries permitted until 2026-07-17 18:53:28.754150675 +0000 UTC m=+534.516580916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q7hdd" (UniqueName: "kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd") pod "kuadrant-operator-controller-manager-86878c769f-5qzjv" (UID: "bfbbeb1d-99a0-4d99-be55-0750cec53033") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-138-140.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object
Jul 17 18:53:28.277130 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.277099    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 18:53:28.280458 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.280434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.303644 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.303595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 18:53:28.352015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.351924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7w5\" (UniqueName: \"kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.352015 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.351987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.453486 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.453449    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7w5\" (UniqueName: \"kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.453663 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.453503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.453870 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.453851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.471555 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.471530    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7w5\" (UniqueName: \"kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5\") pod \"kuadrant-operator-controller-manager-86878c769f-7prvj\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.591035 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.591001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:28.732310 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.732289    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 18:53:28.735367 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:53:28.735339    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98569ae6_f23e_4dd0_bb77_8e430e121ac0.slice/crio-f8d8386935f1d1439f58f05ce71c3ac0c0df092ed0e27b7a1e02b56c90c0e320 WatchSource:0}: Error finding container f8d8386935f1d1439f58f05ce71c3ac0c0df092ed0e27b7a1e02b56c90c0e320: Status 404 returned error can't find the container with id f8d8386935f1d1439f58f05ce71c3ac0c0df092ed0e27b7a1e02b56c90c0e320
Jul 17 18:53:28.756333 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:28.756285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7hdd\" (UniqueName: \"kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd\") pod \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:28.762805 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:28.762789    2558 projected.go:196] Error preparing data for projected volume kube-api-access-q7hdd for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-5qzjv" not found
Jul 17 18:53:28.762867 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:28.762850    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd podName:bfbbeb1d-99a0-4d99-be55-0750cec53033 nodeName:}" failed. No retries permitted until 2026-07-17 18:53:29.762833603 +0000 UTC m=+535.525263841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q7hdd" (UniqueName: "kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd") pod "kuadrant-operator-controller-manager-86878c769f-5qzjv" (UID: "bfbbeb1d-99a0-4d99-be55-0750cec53033") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-5qzjv" not found
Jul 17 18:53:29.107401 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.107360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" event={"ID":"98569ae6-f23e-4dd0-bb77-8e430e121ac0","Type":"ContainerStarted","Data":"f8d8386935f1d1439f58f05ce71c3ac0c0df092ed0e27b7a1e02b56c90c0e320"}
Jul 17 18:53:29.107401 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.107415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:29.115512 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.115490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:29.119487 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:29.119458    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" podUID="bfbbeb1d-99a0-4d99-be55-0750cec53033" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:29.261467 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.261437    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume\") pod \"bfbbeb1d-99a0-4d99-be55-0750cec53033\" (UID: \"bfbbeb1d-99a0-4d99-be55-0750cec53033\") "
Jul 17 18:53:29.261677 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.261658    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7hdd\" (UniqueName: \"kubernetes.io/projected/bfbbeb1d-99a0-4d99-be55-0750cec53033-kube-api-access-q7hdd\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:53:29.261754 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.261731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bfbbeb1d-99a0-4d99-be55-0750cec53033" (UID: "bfbbeb1d-99a0-4d99-be55-0750cec53033"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 18:53:29.362737 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:29.362667    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbbeb1d-99a0-4d99-be55-0750cec53033-extensions-socket-volume\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 18:53:30.112395 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:30.112364    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:30.115268 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:30.115228    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" podUID="bfbbeb1d-99a0-4d99-be55-0750cec53033" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:30.122062 ip-10-0-138-140 kubenswrapper[2558]: E0717 18:53:30.122033    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-5qzjv\" is forbidden: User \"system:node:ip-10-0-138-140.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-140.ec2.internal' and this object" podUID="bfbbeb1d-99a0-4d99-be55-0750cec53033" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5qzjv"
Jul 17 18:53:30.772405 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:30.772374    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbbeb1d-99a0-4d99-be55-0750cec53033" path="/var/lib/kubelet/pods/bfbbeb1d-99a0-4d99-be55-0750cec53033/volumes"
Jul 17 18:53:34.134084 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:34.134049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" event={"ID":"98569ae6-f23e-4dd0-bb77-8e430e121ac0","Type":"ContainerStarted","Data":"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"}
Jul 17 18:53:34.134559 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:34.134205    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:53:34.155821 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:34.155764    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" podStartSLOduration=1.657324935 podStartE2EDuration="6.155742969s" totalImagesPullingTime="4.498418034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:53:34.154605164 +0000 UTC m=+539.917035436" watchObservedRunningTime="2026-07-17 18:53:34.155742969 +0000 UTC m=+539.918173231"
Jul 17 18:53:45.141842 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:53:45.141812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 18:54:34.689108 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:54:34.689083    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:54:34.693930 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:54:34.693907    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:55:37.646382 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.646340    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"]
Jul 17 18:55:37.649676 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.649657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.652678 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.652660    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 18:55:37.652799 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.652720    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 18:55:37.653636 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.653620    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 18:55:37.653876 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.653859    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tmjdx\""
Jul 17 18:55:37.669623 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.669601    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"]
Jul 17 18:55:37.698787 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.698759    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.698904 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.698806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcv6\" (UniqueName: \"kubernetes.io/projected/be63392d-9c1b-43f2-8af2-96f4d39b5c67-kube-api-access-9xcv6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.698974 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.698908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.698974 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.698964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.699084 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.699003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.699084 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.699031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799518 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799699 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcv6\" (UniqueName: \"kubernetes.io/projected/be63392d-9c1b-43f2-8af2-96f4d39b5c67-kube-api-access-9xcv6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799699 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799699 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799699 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.799897 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.799759    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.800034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.800003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.800034 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.800029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.800136 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.800097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.801915 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.801896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be63392d-9c1b-43f2-8af2-96f4d39b5c67-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.802236 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.802213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be63392d-9c1b-43f2-8af2-96f4d39b5c67-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.811126 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.811101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcv6\" (UniqueName: \"kubernetes.io/projected/be63392d-9c1b-43f2-8af2-96f4d39b5c67-kube-api-access-9xcv6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9\" (UID: \"be63392d-9c1b-43f2-8af2-96f4d39b5c67\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:37.960208 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:37.960128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:38.098383 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:38.098352    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"]
Jul 17 18:55:38.100788 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:55:38.100761    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe63392d_9c1b_43f2_8af2_96f4d39b5c67.slice/crio-720e6108aa5feced3e9998e81026ea7051b704f16f571a3e75f4a09c338c00dd WatchSource:0}: Error finding container 720e6108aa5feced3e9998e81026ea7051b704f16f571a3e75f4a09c338c00dd: Status 404 returned error can't find the container with id 720e6108aa5feced3e9998e81026ea7051b704f16f571a3e75f4a09c338c00dd
Jul 17 18:55:38.677152 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:38.677109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9" event={"ID":"be63392d-9c1b-43f2-8af2-96f4d39b5c67","Type":"ContainerStarted","Data":"720e6108aa5feced3e9998e81026ea7051b704f16f571a3e75f4a09c338c00dd"}
Jul 17 18:55:39.682338 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:39.682289    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9" event={"ID":"be63392d-9c1b-43f2-8af2-96f4d39b5c67","Type":"ContainerStarted","Data":"f60f066617f2c76618fad9e0e7c54fbafc23d41d5bc3985795f7e449abe53af5"}
Jul 17 18:55:39.705834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:39.705786    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9" podStartSLOduration=1.576960781 podStartE2EDuration="2.705769209s" totalImagesPullingTime="1.128808428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:55:39.704107286 +0000 UTC m=+665.466537566" watchObservedRunningTime="2026-07-17 18:55:39.705769209 +0000 UTC m=+665.468199527"
Jul 17 18:55:47.961175 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:47.961137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:47.961714 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:47.961218    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:47.973503 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:47.973476    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:55:48.733973 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:55:48.733947    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9"
Jul 17 18:56:04.554291 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.554262    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"]
Jul 17 18:56:04.557567 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.557550    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.560511 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.560494    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 18:56:04.570595 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.570575    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"]
Jul 17 18:56:04.746292 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.746292 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746294    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.746526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.746526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.746526 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746482    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.746620 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.746557    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfs7\" (UniqueName: \"kubernetes.io/projected/3ec2fe88-76e6-4d17-9a6f-4aee17650144-kube-api-access-dgfs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847302 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847407    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfs7\" (UniqueName: \"kubernetes.io/projected/3ec2fe88-76e6-4d17-9a6f-4aee17650144-kube-api-access-dgfs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847657 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847821 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.847821 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.847773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.849791 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.849769    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec2fe88-76e6-4d17-9a6f-4aee17650144-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.849952 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.849932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec2fe88-76e6-4d17-9a6f-4aee17650144-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.857026 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.857005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfs7\" (UniqueName: \"kubernetes.io/projected/3ec2fe88-76e6-4d17-9a6f-4aee17650144-kube-api-access-dgfs7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-56zdh\" (UID: \"3ec2fe88-76e6-4d17-9a6f-4aee17650144\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:04.867834 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:04.867808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:05.012943 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:05.012920    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"]
Jul 17 18:56:05.015213 ip-10-0-138-140 kubenswrapper[2558]: W0717 18:56:05.015187    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec2fe88_76e6_4d17_9a6f_4aee17650144.slice/crio-1a96668e4b70b1ba4f3f2eb63eb1404fa415307686430f60d1f37e8c483d9f46 WatchSource:0}: Error finding container 1a96668e4b70b1ba4f3f2eb63eb1404fa415307686430f60d1f37e8c483d9f46: Status 404 returned error can't find the container with id 1a96668e4b70b1ba4f3f2eb63eb1404fa415307686430f60d1f37e8c483d9f46
Jul 17 18:56:05.795150 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:05.795116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh" event={"ID":"3ec2fe88-76e6-4d17-9a6f-4aee17650144","Type":"ContainerStarted","Data":"965cd9b4bfd12cd8b623de9a7696577249ad257f63d8c5f65729bd94b9e141f7"}
Jul 17 18:56:05.795589 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:05.795178    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh" event={"ID":"3ec2fe88-76e6-4d17-9a6f-4aee17650144","Type":"ContainerStarted","Data":"1a96668e4b70b1ba4f3f2eb63eb1404fa415307686430f60d1f37e8c483d9f46"}
Jul 17 18:56:05.817938 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:05.817872    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh" podStartSLOduration=1.634261169 podStartE2EDuration="1.81785768s" totalImagesPullingTime="183.596511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:05.815902877 +0000 UTC m=+691.578333136" watchObservedRunningTime="2026-07-17 18:56:05.81785768 +0000 UTC m=+691.580287988"
Jul 17 18:56:14.868580 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:14.868539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:14.869042 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:14.868620    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:14.881208 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:14.881183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:56:15.849436 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:56:15.849399    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-56zdh"
Jul 17 18:59:34.717558 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:59:34.717489    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 18:59:34.719439 ip-10-0-138-140 kubenswrapper[2558]: I0717 18:59:34.719421    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:00:00.147716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.147680    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"]
Jul 17 19:00:00.151031 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.151014    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"
Jul 17 19:00:00.153707 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.153685    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:00:00.157576 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.157554    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"]
Jul 17 19:00:00.161875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.161853    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"
Jul 17 19:00:00.303501 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.303471    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"]
Jul 17 19:00:00.308133 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:00:00.308112    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b222375_a939_4653_b8f9_992a895bb941.slice/crio-fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd WatchSource:0}: Error finding container fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd: Status 404 returned error can't find the container with id fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd
Jul 17 19:00:00.310003 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.309989    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:00:00.845840 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:00.845808    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc" event={"ID":"0b222375-a939-4653-b8f9-992a895bb941","Type":"ContainerStarted","Data":"fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd"}
Jul 17 19:00:05.871811 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:05.871776    2558 generic.go:356] "Generic (PLEG): container finished" podID="0b222375-a939-4653-b8f9-992a895bb941" containerID="cadbb522806f54d926905fc10cd12a215bcd995cdb001fd943656ae5e8854cd5" exitCode=0
Jul 17 19:00:05.872233 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:05.871848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc" event={"ID":"0b222375-a939-4653-b8f9-992a895bb941","Type":"ContainerDied","Data":"cadbb522806f54d926905fc10cd12a215bcd995cdb001fd943656ae5e8854cd5"}
Jul 17 19:00:07.008428 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:07.008401    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"
Jul 17 19:00:07.882554 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:07.882524    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc"
Jul 17 19:00:07.882729 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:07.882523    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738580-bpzlc" event={"ID":"0b222375-a939-4653-b8f9-992a895bb941","Type":"ContainerDied","Data":"fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd"}
Jul 17 19:00:07.882729 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:00:07.882643    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc78d1fcc267b5ff493bd902dbc009e0ca36d82cdcdd9c7319cda3197473dedd"
Jul 17 19:04:34.745930 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:34.745904    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:04:34.746513 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:34.746148    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:04:47.507375 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:47.507315    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 19:04:47.507823 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:47.507581    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" podUID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" containerName="manager" containerID="cri-o://0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912" gracePeriod=10
Jul 17 19:04:50.069114 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.069086    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 19:04:50.140626 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.140587    2558 generic.go:356] "Generic (PLEG): container finished" podID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" containerID="0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912" exitCode=0
Jul 17 19:04:50.140805 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.140650    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"
Jul 17 19:04:50.140805 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.140676    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" event={"ID":"98569ae6-f23e-4dd0-bb77-8e430e121ac0","Type":"ContainerDied","Data":"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"}
Jul 17 19:04:50.140805 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.140757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj" event={"ID":"98569ae6-f23e-4dd0-bb77-8e430e121ac0","Type":"ContainerDied","Data":"f8d8386935f1d1439f58f05ce71c3ac0c0df092ed0e27b7a1e02b56c90c0e320"}
Jul 17 19:04:50.140805 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.140780    2558 scope.go:117] "RemoveContainer" containerID="0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"
Jul 17 19:04:50.152206 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.152183    2558 scope.go:117] "RemoveContainer" containerID="0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"
Jul 17 19:04:50.152947 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:04:50.152922    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912\": container with ID starting with 0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912 not found: ID does not exist" containerID="0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"
Jul 17 19:04:50.153034 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.152960    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912"} err="failed to get container status \"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912\": rpc error: code = NotFound desc = could not find container \"0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912\": container with ID starting with 0896b9e488753686ced23f580da21365a5134c539b2456ee9eb98d6c9d305912 not found: ID does not exist"
Jul 17 19:04:50.154183 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.154166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv7w5\" (UniqueName: \"kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5\") pod \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") "
Jul 17 19:04:50.154244 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.154218    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume\") pod \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\" (UID: \"98569ae6-f23e-4dd0-bb77-8e430e121ac0\") "
Jul 17 19:04:50.154605 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.154583    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "98569ae6-f23e-4dd0-bb77-8e430e121ac0" (UID: "98569ae6-f23e-4dd0-bb77-8e430e121ac0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:04:50.156485 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.156466    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5" (OuterVolumeSpecName: "kube-api-access-mv7w5") pod "98569ae6-f23e-4dd0-bb77-8e430e121ac0" (UID: "98569ae6-f23e-4dd0-bb77-8e430e121ac0"). InnerVolumeSpecName "kube-api-access-mv7w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:04:50.255384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.255348    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7w5\" (UniqueName: \"kubernetes.io/projected/98569ae6-f23e-4dd0-bb77-8e430e121ac0-kube-api-access-mv7w5\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:04:50.255384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.255379    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/98569ae6-f23e-4dd0-bb77-8e430e121ac0-extensions-socket-volume\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:04:50.464341 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.464290    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 19:04:50.469002 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.468976    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7prvj"]
Jul 17 19:04:50.773515 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:04:50.773425    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" path="/var/lib/kubelet/pods/98569ae6-f23e-4dd0-bb77-8e430e121ac0/volumes"
Jul 17 19:05:48.657276 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657237    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"]
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657590    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" containerName="manager"
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657601    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" containerName="manager"
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657625    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b222375-a939-4653-b8f9-992a895bb941" containerName="cleanup"
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657630    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b222375-a939-4653-b8f9-992a895bb941" containerName="cleanup"
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657678    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="98569ae6-f23e-4dd0-bb77-8e430e121ac0" containerName="manager"
Jul 17 19:05:48.657737 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.657688    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b222375-a939-4653-b8f9-992a895bb941" containerName="cleanup"
Jul 17 19:05:48.662170 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.662150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.664845 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.664826    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 19:05:48.664960 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.664862    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 19:05:48.664960 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.664890    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sfrlg\""
Jul 17 19:05:48.673462 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.673435    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"]
Jul 17 19:05:48.693097 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.693070    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgq75\" (UniqueName: \"kubernetes.io/projected/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-kube-api-access-vgq75\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.693367 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.693194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.794230 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.794198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.794414 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.794242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgq75\" (UniqueName: \"kubernetes.io/projected/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-kube-api-access-vgq75\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.794585 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.794562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.804331 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.804303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgq75\" (UniqueName: \"kubernetes.io/projected/c1ed5bae-be7f-43a1-9253-2ad684ddc5d4-kube-api-access-vgq75\") pod \"kuadrant-operator-controller-manager-86878c769f-cgh2f\" (UID: \"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:48.972781 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:48.972695    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:49.121613 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:49.121578    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"]
Jul 17 19:05:49.128018 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:05:49.126430    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ed5bae_be7f_43a1_9253_2ad684ddc5d4.slice/crio-c73800ccda72cefd3a6d0477cb0a69c204d22c27f356c2a1da614fda311120a0 WatchSource:0}: Error finding container c73800ccda72cefd3a6d0477cb0a69c204d22c27f356c2a1da614fda311120a0: Status 404 returned error can't find the container with id c73800ccda72cefd3a6d0477cb0a69c204d22c27f356c2a1da614fda311120a0
Jul 17 19:05:49.414569 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:49.414517    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f" event={"ID":"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4","Type":"ContainerStarted","Data":"b887fc6ad96b0c37dacc72116871e16c721a5cab735132b37c8b1206031b2fbd"}
Jul 17 19:05:49.414730 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:49.414607    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f" event={"ID":"c1ed5bae-be7f-43a1-9253-2ad684ddc5d4","Type":"ContainerStarted","Data":"c73800ccda72cefd3a6d0477cb0a69c204d22c27f356c2a1da614fda311120a0"}
Jul 17 19:05:49.414730 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:49.414643    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:05:49.438739 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:05:49.438687    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f" podStartSLOduration=1.438668513 podStartE2EDuration="1.438668513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:05:49.436980517 +0000 UTC m=+1275.199410778" watchObservedRunningTime="2026-07-17 19:05:49.438668513 +0000 UTC m=+1275.201098772"
Jul 17 19:06:00.422490 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:06:00.422454    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cgh2f"
Jul 17 19:09:34.772390 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:09:34.772361    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:09:34.773423 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:09:34.773401    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:12:47.920541 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:47.920503    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"]
Jul 17 19:12:47.923902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:47.923885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:47.926750 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:47.926729    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:12:47.926872 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:47.926729    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:12:47.933517 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:47.933496    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"]
Jul 17 19:12:48.023768 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:48.023710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp\" (UID: \"95807983-418b-4162-9bc1-5264587bc366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:48.125275 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:48.125219    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp\" (UID: \"95807983-418b-4162-9bc1-5264587bc366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:48.125867 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:48.125845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp\" (UID: \"95807983-418b-4162-9bc1-5264587bc366\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:48.234478 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:48.234390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:48.374472 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:48.374448    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"]
Jul 17 19:12:48.376412 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:12:48.376387    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95807983_418b_4162_9bc1_5264587bc366.slice/crio-b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e WatchSource:0}: Error finding container b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e: Status 404 returned error can't find the container with id b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e
Jul 17 19:12:49.300548 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:49.300519    2558 generic.go:356] "Generic (PLEG): container finished" podID="95807983-418b-4162-9bc1-5264587bc366" containerID="2b5470b9c940b9269c89edd9a1fd9652e32856de2eaa5ae27754dad7031193b9" exitCode=0
Jul 17 19:12:49.300997 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:49.300605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp" event={"ID":"95807983-418b-4162-9bc1-5264587bc366","Type":"ContainerDied","Data":"2b5470b9c940b9269c89edd9a1fd9652e32856de2eaa5ae27754dad7031193b9"}
Jul 17 19:12:49.300997 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:49.300663    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp" event={"ID":"95807983-418b-4162-9bc1-5264587bc366","Type":"ContainerStarted","Data":"b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e"}
Jul 17 19:12:50.439706 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:50.439682    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:50.447370 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:50.447341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca\") pod \"95807983-418b-4162-9bc1-5264587bc366\" (UID: \"95807983-418b-4162-9bc1-5264587bc366\") "
Jul 17 19:12:50.447694 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:50.447672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "95807983-418b-4162-9bc1-5264587bc366" (UID: "95807983-418b-4162-9bc1-5264587bc366"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:12:50.548502 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:50.548459    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95807983-418b-4162-9bc1-5264587bc366-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:12:51.311345 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:51.311290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp" event={"ID":"95807983-418b-4162-9bc1-5264587bc366","Type":"ContainerDied","Data":"b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e"}
Jul 17 19:12:51.311517 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:51.311354    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"
Jul 17 19:12:51.311517 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:12:51.311379    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e8d704204ac81957c701edba856d61f4ae5a431560e19037a03073c1ee403e"
Jul 17 19:13:03.045110 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:03.045074    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"]
Jul 17 19:13:03.050199 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:03.050178    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb2e5b30-qshlp"]
Jul 17 19:13:04.772544 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:04.772515    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95807983-418b-4162-9bc1-5264587bc366" path="/var/lib/kubelet/pods/95807983-418b-4162-9bc1-5264587bc366/volumes"
Jul 17 19:13:06.848242 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.848203    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:06.848716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.848693    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95807983-418b-4162-9bc1-5264587bc366" containerName="revoke-keys"
Jul 17 19:13:06.848716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.848710    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="95807983-418b-4162-9bc1-5264587bc366" containerName="revoke-keys"
Jul 17 19:13:06.848829 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.848808    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="95807983-418b-4162-9bc1-5264587bc366" containerName="revoke-keys"
Jul 17 19:13:06.852901 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.852876    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:06.858183 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.858155    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-eb352e55-serving-cert\""
Jul 17 19:13:06.858310 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.858209    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:13:06.859296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.859272    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:13:06.859296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.859295    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:13:06.866202 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.866163    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:06.899657 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.899620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:06.899847 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.899753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qw2t\" (UniqueName: \"kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:06.899847 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:06.899803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.000276 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.000247    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qw2t\" (UniqueName: \"kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.000471 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.000291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.000471 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.000414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.001035 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.001013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.002741 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.002712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.020051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.020029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qw2t\" (UniqueName: \"kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t\") pod \"maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.165532 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.165443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:07.312125 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.312101    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:07.317724 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:13:07.317694    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd223ab_6694_47b7_b569_c676feb0b5a5.slice/crio-5113fe34db6f31e0702370ed937ed1d88a256c9ca1d921ce39f95ed40578721d WatchSource:0}: Error finding container 5113fe34db6f31e0702370ed937ed1d88a256c9ca1d921ce39f95ed40578721d: Status 404 returned error can't find the container with id 5113fe34db6f31e0702370ed937ed1d88a256c9ca1d921ce39f95ed40578721d
Jul 17 19:13:07.319548 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.319523    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:13:07.384764 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:07.384732    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" event={"ID":"bdd223ab-6694-47b7-b569-c676feb0b5a5","Type":"ContainerStarted","Data":"5113fe34db6f31e0702370ed937ed1d88a256c9ca1d921ce39f95ed40578721d"}
Jul 17 19:13:10.402157 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:10.402111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" event={"ID":"bdd223ab-6694-47b7-b569-c676feb0b5a5","Type":"ContainerStarted","Data":"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"}
Jul 17 19:13:10.402537 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:10.402259    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:10.421453 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:10.421399    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" podStartSLOduration=2.10341333 podStartE2EDuration="4.421381747s" totalImagesPullingTime="2.317968417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:13:10.419466425 +0000 UTC m=+1716.181896686" watchObservedRunningTime="2026-07-17 19:13:10.421381747 +0000 UTC m=+1716.183812006"
Jul 17 19:13:16.413809 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:16.413780    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:31.687451 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.687420    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"]
Jul 17 19:13:31.690963 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.690943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:31.693751 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.693731    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:13:31.701149 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.701125    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"]
Jul 17 19:13:31.740887 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.740856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9\" (UID: \"6b14073d-658c-49ab-a0e5-5522bc9d2174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:31.842007 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.841956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9\" (UID: \"6b14073d-658c-49ab-a0e5-5522bc9d2174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:31.842632 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:31.842611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9\" (UID: \"6b14073d-658c-49ab-a0e5-5522bc9d2174\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:32.001966 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:32.001879    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:32.138463 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:32.138440    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"]
Jul 17 19:13:32.140678 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:13:32.140648    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b14073d_658c_49ab_a0e5_5522bc9d2174.slice/crio-fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea WatchSource:0}: Error finding container fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea: Status 404 returned error can't find the container with id fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea
Jul 17 19:13:32.510744 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:32.510715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9" event={"ID":"6b14073d-658c-49ab-a0e5-5522bc9d2174","Type":"ContainerStarted","Data":"37d14b7dfb2fcfe4c7c22f3e527acc44ffa83aa259538f13d74688340f61ea34"}
Jul 17 19:13:32.510912 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:32.510762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9" event={"ID":"6b14073d-658c-49ab-a0e5-5522bc9d2174","Type":"ContainerStarted","Data":"fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea"}
Jul 17 19:13:32.534793 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:32.534746    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9" podStartSLOduration=1.534731887 podStartE2EDuration="1.534731887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:13:32.534572833 +0000 UTC m=+1738.297003094" watchObservedRunningTime="2026-07-17 19:13:32.534731887 +0000 UTC m=+1738.297162146"
Jul 17 19:13:33.516786 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:33.516748    2558 generic.go:356] "Generic (PLEG): container finished" podID="6b14073d-658c-49ab-a0e5-5522bc9d2174" containerID="37d14b7dfb2fcfe4c7c22f3e527acc44ffa83aa259538f13d74688340f61ea34" exitCode=0
Jul 17 19:13:33.517181 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:33.516813    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9" event={"ID":"6b14073d-658c-49ab-a0e5-5522bc9d2174","Type":"ContainerDied","Data":"37d14b7dfb2fcfe4c7c22f3e527acc44ffa83aa259538f13d74688340f61ea34"}
Jul 17 19:13:34.648816 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:34.648791    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:34.773833 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:34.773754    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca\") pod \"6b14073d-658c-49ab-a0e5-5522bc9d2174\" (UID: \"6b14073d-658c-49ab-a0e5-5522bc9d2174\") "
Jul 17 19:13:34.774087 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:34.774063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b14073d-658c-49ab-a0e5-5522bc9d2174" (UID: "6b14073d-658c-49ab-a0e5-5522bc9d2174"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:34.874570 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:34.874541    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b14073d-658c-49ab-a0e5-5522bc9d2174-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:13:35.527272 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:35.527243    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"
Jul 17 19:13:35.527272 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:35.527262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9" event={"ID":"6b14073d-658c-49ab-a0e5-5522bc9d2174","Type":"ContainerDied","Data":"fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea"}
Jul 17 19:13:35.527541 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:35.527304    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3a2a6d6c0dc5e0e94b554321b00ab9de91d322cfb557be532aa9f15afaebea"
Jul 17 19:13:42.264052 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.264016    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:42.264573 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.264266    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" podUID="bdd223ab-6694-47b7-b569-c676feb0b5a5" containerName="maas-api" containerID="cri-o://8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d" gracePeriod=30
Jul 17 19:13:42.523471 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.523410    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:42.560638 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.560604    2558 generic.go:356] "Generic (PLEG): container finished" podID="bdd223ab-6694-47b7-b569-c676feb0b5a5" containerID="8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d" exitCode=0
Jul 17 19:13:42.560638 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.560644    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"
Jul 17 19:13:42.560902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.560686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" event={"ID":"bdd223ab-6694-47b7-b569-c676feb0b5a5","Type":"ContainerDied","Data":"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"}
Jul 17 19:13:42.560902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.560745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr" event={"ID":"bdd223ab-6694-47b7-b569-c676feb0b5a5","Type":"ContainerDied","Data":"5113fe34db6f31e0702370ed937ed1d88a256c9ca1d921ce39f95ed40578721d"}
Jul 17 19:13:42.560902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.560774    2558 scope.go:117] "RemoveContainer" containerID="8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"
Jul 17 19:13:42.569651 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.569628    2558 scope.go:117] "RemoveContainer" containerID="8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"
Jul 17 19:13:42.569984 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:13:42.569963    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d\": container with ID starting with 8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d not found: ID does not exist" containerID="8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"
Jul 17 19:13:42.570052 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.569995    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d"} err="failed to get container status \"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d\": rpc error: code = NotFound desc = could not find container \"8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d\": container with ID starting with 8ee1ecfc05f32d4be36ab2334a39b8cd1fda5b0861fbe0cbec478cf43163ae9d not found: ID does not exist"
Jul 17 19:13:42.650291 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.650257    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qw2t\" (UniqueName: \"kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t\") pod \"bdd223ab-6694-47b7-b569-c676feb0b5a5\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") "
Jul 17 19:13:42.650493 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.650346    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls\") pod \"bdd223ab-6694-47b7-b569-c676feb0b5a5\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") "
Jul 17 19:13:42.650493 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.650374    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca\") pod \"bdd223ab-6694-47b7-b569-c676feb0b5a5\" (UID: \"bdd223ab-6694-47b7-b569-c676feb0b5a5\") "
Jul 17 19:13:42.650776 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.650752    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdd223ab-6694-47b7-b569-c676feb0b5a5" (UID: "bdd223ab-6694-47b7-b569-c676feb0b5a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:13:42.652656 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.652632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t" (OuterVolumeSpecName: "kube-api-access-4qw2t") pod "bdd223ab-6694-47b7-b569-c676feb0b5a5" (UID: "bdd223ab-6694-47b7-b569-c676feb0b5a5"). InnerVolumeSpecName "kube-api-access-4qw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:13:42.652756 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.652707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdd223ab-6694-47b7-b569-c676feb0b5a5" (UID: "bdd223ab-6694-47b7-b569-c676feb0b5a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:13:42.751803 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.751772    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qw2t\" (UniqueName: \"kubernetes.io/projected/bdd223ab-6694-47b7-b569-c676feb0b5a5-kube-api-access-4qw2t\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:13:42.751803 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.751800    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd223ab-6694-47b7-b569-c676feb0b5a5-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:13:42.751803 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.751808    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd223ab-6694-47b7-b569-c676feb0b5a5-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:13:42.879911 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.879774    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:42.881415 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:42.881391    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-eb352e55-7f8f74d4b5-tjcnr"]
Jul 17 19:13:44.773941 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:44.773904    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd223ab-6694-47b7-b569-c676feb0b5a5" path="/var/lib/kubelet/pods/bdd223ab-6694-47b7-b569-c676feb0b5a5/volumes"
Jul 17 19:13:46.821724 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:46.821686    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"]
Jul 17 19:13:46.830192 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:46.830166    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-eb352e55-b8rl9"]
Jul 17 19:13:48.772616 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:48.772583    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b14073d-658c-49ab-a0e5-5522bc9d2174" path="/var/lib/kubelet/pods/6b14073d-658c-49ab-a0e5-5522bc9d2174/volumes"
Jul 17 19:13:50.084690 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.084646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:13:50.085132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085049    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd223ab-6694-47b7-b569-c676feb0b5a5" containerName="maas-api"
Jul 17 19:13:50.085132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085062    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd223ab-6694-47b7-b569-c676feb0b5a5" containerName="maas-api"
Jul 17 19:13:50.085132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085084    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b14073d-658c-49ab-a0e5-5522bc9d2174" containerName="revoke-keys"
Jul 17 19:13:50.085132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085090    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b14073d-658c-49ab-a0e5-5522bc9d2174" containerName="revoke-keys"
Jul 17 19:13:50.085313 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085146    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b14073d-658c-49ab-a0e5-5522bc9d2174" containerName="revoke-keys"
Jul 17 19:13:50.085313 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.085157    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd223ab-6694-47b7-b569-c676feb0b5a5" containerName="maas-api"
Jul 17 19:13:50.096532 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.096504    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.100005 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.099977    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e544e90b-serving-cert\""
Jul 17 19:13:50.100292 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.100275    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:13:50.101212 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.101175    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:13:50.102933 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.101910    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:13:50.104280 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.104217    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:13:50.221339 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.221286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.221497 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.221367    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.221497 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.221415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.322851 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.322815    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.322851 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.322861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.323138 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.323063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.323138 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:13:50.323062    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-serving-cert: secret "maas-api-e2e-ait-e544e90b-serving-cert" not found
Jul 17 19:13:50.323261 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:13:50.323142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls podName:91d4b362-8f1a-4dd0-a263-f805da2eb195 nodeName:}" failed. No retries permitted until 2026-07-17 19:13:50.823120796 +0000 UTC m=+1756.585551033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls") pod "maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" (UID: "91d4b362-8f1a-4dd0-a263-f805da2eb195") : secret "maas-api-e2e-ait-e544e90b-serving-cert" not found
Jul 17 19:13:50.323625 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.323601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.344979 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.344909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.828305 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.828275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:50.830838 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:50.830815    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") pod \"maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:51.010810 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:51.010768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:51.173025 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:51.172993    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:13:51.176799 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:13:51.176770    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d4b362_8f1a_4dd0_a263_f805da2eb195.slice/crio-f145fceb84d708089b3f35e47fb13b570d6258944f76161aa0fd7d21e0aab60e WatchSource:0}: Error finding container f145fceb84d708089b3f35e47fb13b570d6258944f76161aa0fd7d21e0aab60e: Status 404 returned error can't find the container with id f145fceb84d708089b3f35e47fb13b570d6258944f76161aa0fd7d21e0aab60e
Jul 17 19:13:51.605019 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:51.604983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" event={"ID":"91d4b362-8f1a-4dd0-a263-f805da2eb195","Type":"ContainerStarted","Data":"f145fceb84d708089b3f35e47fb13b570d6258944f76161aa0fd7d21e0aab60e"}
Jul 17 19:13:52.610866 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:52.610827    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" event={"ID":"91d4b362-8f1a-4dd0-a263-f805da2eb195","Type":"ContainerStarted","Data":"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"}
Jul 17 19:13:52.611311 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:52.610980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:13:52.644307 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:52.644252    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" podStartSLOduration=2.2810887859999998 podStartE2EDuration="2.644236567s" totalImagesPullingTime="363.147781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:13:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:13:52.643536034 +0000 UTC m=+1758.405966294" watchObservedRunningTime="2026-07-17 19:13:52.644236567 +0000 UTC m=+1758.406666828"
Jul 17 19:13:58.621124 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:13:58.621091    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:14:15.571813 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.571774    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"]
Jul 17 19:14:15.575262 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.575245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:15.577859 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.577840    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:14:15.584070 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.584043    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"]
Jul 17 19:14:15.655310 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.655273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6\" (UID: \"76bd510e-f424-460d-b53d-719b7906ffba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:15.756792 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.756762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6\" (UID: \"76bd510e-f424-460d-b53d-719b7906ffba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:15.757365 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.757348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6\" (UID: \"76bd510e-f424-460d-b53d-719b7906ffba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:15.887774 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:15.887690    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:16.023875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:16.023845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"]
Jul 17 19:14:16.026783 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:14:16.026756    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bd510e_f424_460d_b53d_719b7906ffba.slice/crio-b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92 WatchSource:0}: Error finding container b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92: Status 404 returned error can't find the container with id b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92
Jul 17 19:14:16.724376 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:16.724344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6" event={"ID":"76bd510e-f424-460d-b53d-719b7906ffba","Type":"ContainerStarted","Data":"3aee3d838a827bfcd4393d4ff728f0c9ad2d7e80d1c071a1b67d6b6ab76c23c2"}
Jul 17 19:14:16.724774 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:16.724394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6" event={"ID":"76bd510e-f424-460d-b53d-719b7906ffba","Type":"ContainerStarted","Data":"b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92"}
Jul 17 19:14:16.745430 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:16.745380    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6" podStartSLOduration=1.745341545 podStartE2EDuration="1.745341545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:14:16.743810758 +0000 UTC m=+1782.506241020" watchObservedRunningTime="2026-07-17 19:14:16.745341545 +0000 UTC m=+1782.507771796"
Jul 17 19:14:17.736529 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:17.736488    2558 generic.go:356] "Generic (PLEG): container finished" podID="76bd510e-f424-460d-b53d-719b7906ffba" containerID="3aee3d838a827bfcd4393d4ff728f0c9ad2d7e80d1c071a1b67d6b6ab76c23c2" exitCode=0
Jul 17 19:14:17.736954 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:17.736572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6" event={"ID":"76bd510e-f424-460d-b53d-719b7906ffba","Type":"ContainerDied","Data":"3aee3d838a827bfcd4393d4ff728f0c9ad2d7e80d1c071a1b67d6b6ab76c23c2"}
Jul 17 19:14:18.876910 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:18.876886    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:18.989804 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:18.989773    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca\") pod \"76bd510e-f424-460d-b53d-719b7906ffba\" (UID: \"76bd510e-f424-460d-b53d-719b7906ffba\") "
Jul 17 19:14:18.990185 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:18.990158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76bd510e-f424-460d-b53d-719b7906ffba" (UID: "76bd510e-f424-460d-b53d-719b7906ffba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:19.090787 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:19.090761    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76bd510e-f424-460d-b53d-719b7906ffba-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:14:19.747284 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:19.747256    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"
Jul 17 19:14:19.747284 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:19.747277    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6" event={"ID":"76bd510e-f424-460d-b53d-719b7906ffba","Type":"ContainerDied","Data":"b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92"}
Jul 17 19:14:19.747510 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:19.747346    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9bc51cdee1bc423a5f75f725d2edb8bb9ff1ed6818a00fabf20c03703988b92"
Jul 17 19:14:26.267232 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.267154    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:14:26.267664 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.267417    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" podUID="91d4b362-8f1a-4dd0-a263-f805da2eb195" containerName="maas-api" containerID="cri-o://00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d" gracePeriod=30
Jul 17 19:14:26.515979 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.515953    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:14:26.658630 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.658593    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4\") pod \"91d4b362-8f1a-4dd0-a263-f805da2eb195\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") "
Jul 17 19:14:26.658818 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.658647    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca\") pod \"91d4b362-8f1a-4dd0-a263-f805da2eb195\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") "
Jul 17 19:14:26.658818 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.658799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") pod \"91d4b362-8f1a-4dd0-a263-f805da2eb195\" (UID: \"91d4b362-8f1a-4dd0-a263-f805da2eb195\") "
Jul 17 19:14:26.659071 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.659044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91d4b362-8f1a-4dd0-a263-f805da2eb195" (UID: "91d4b362-8f1a-4dd0-a263-f805da2eb195"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:14:26.660986 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.660959    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91d4b362-8f1a-4dd0-a263-f805da2eb195" (UID: "91d4b362-8f1a-4dd0-a263-f805da2eb195"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:14:26.660986 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.660965    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4" (OuterVolumeSpecName: "kube-api-access-crhk4") pod "91d4b362-8f1a-4dd0-a263-f805da2eb195" (UID: "91d4b362-8f1a-4dd0-a263-f805da2eb195"). InnerVolumeSpecName "kube-api-access-crhk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:14:26.760468 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.760434    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d4b362-8f1a-4dd0-a263-f805da2eb195-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:14:26.760468 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.760462    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhk4\" (UniqueName: \"kubernetes.io/projected/91d4b362-8f1a-4dd0-a263-f805da2eb195-kube-api-access-crhk4\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:14:26.760468 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.760472    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d4b362-8f1a-4dd0-a263-f805da2eb195-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:14:26.789477 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.789450    2558 generic.go:356] "Generic (PLEG): container finished" podID="91d4b362-8f1a-4dd0-a263-f805da2eb195" containerID="00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d" exitCode=0
Jul 17 19:14:26.789606 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.789489    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"
Jul 17 19:14:26.789606 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.789526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" event={"ID":"91d4b362-8f1a-4dd0-a263-f805da2eb195","Type":"ContainerDied","Data":"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"}
Jul 17 19:14:26.789606 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.789565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg" event={"ID":"91d4b362-8f1a-4dd0-a263-f805da2eb195","Type":"ContainerDied","Data":"f145fceb84d708089b3f35e47fb13b570d6258944f76161aa0fd7d21e0aab60e"}
Jul 17 19:14:26.789606 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.789586    2558 scope.go:117] "RemoveContainer" containerID="00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"
Jul 17 19:14:26.798386 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.798364    2558 scope.go:117] "RemoveContainer" containerID="00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"
Jul 17 19:14:26.798663 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:14:26.798645    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d\": container with ID starting with 00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d not found: ID does not exist" containerID="00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"
Jul 17 19:14:26.798714 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.798671    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d"} err="failed to get container status \"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d\": rpc error: code = NotFound desc = could not find container \"00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d\": container with ID starting with 00980fb27697f9779984beef06d16e87d6462b69e7137f4ff46980d6c8579d5d not found: ID does not exist"
Jul 17 19:14:26.820092 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.820067    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:14:26.824366 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:26.824342    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e544e90b-65dc84bfb8-pknvg"]
Jul 17 19:14:28.773666 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:28.773628    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d4b362-8f1a-4dd0-a263-f805da2eb195" path="/var/lib/kubelet/pods/91d4b362-8f1a-4dd0-a263-f805da2eb195/volumes"
Jul 17 19:14:30.674342 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:30.674290    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"]
Jul 17 19:14:30.680085 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:30.680060    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e544e90b-2mbf6"]
Jul 17 19:14:30.774400 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:30.774370    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76bd510e-f424-460d-b53d-719b7906ffba" path="/var/lib/kubelet/pods/76bd510e-f424-460d-b53d-719b7906ffba/volumes"
Jul 17 19:14:34.807337 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:34.807284    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:14:34.808811 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:34.808790    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:14:35.163649 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.163574    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:14:35.163933 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.163921    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76bd510e-f424-460d-b53d-719b7906ffba" containerName="revoke-keys"
Jul 17 19:14:35.163980 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.163935    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="76bd510e-f424-460d-b53d-719b7906ffba" containerName="revoke-keys"
Jul 17 19:14:35.163980 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.163954    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d4b362-8f1a-4dd0-a263-f805da2eb195" containerName="maas-api"
Jul 17 19:14:35.163980 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.163959    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d4b362-8f1a-4dd0-a263-f805da2eb195" containerName="maas-api"
Jul 17 19:14:35.164086 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.164010    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d4b362-8f1a-4dd0-a263-f805da2eb195" containerName="maas-api"
Jul 17 19:14:35.164086 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.164019    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="76bd510e-f424-460d-b53d-719b7906ffba" containerName="revoke-keys"
Jul 17 19:14:35.168370 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.168351    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.171074 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.171052    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:14:35.171200 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.171052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a8061117-serving-cert\""
Jul 17 19:14:35.171200 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.171116    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:14:35.172075 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.172060    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:14:35.193923 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.193898    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:14:35.342006 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.341968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.342184 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.342010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.342184 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.342116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb7bl\" (UniqueName: \"kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.443081 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.442997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb7bl\" (UniqueName: \"kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.443081 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.443074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.443278 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.443105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.443885 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.443859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.445782 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.445761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.455869 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.455847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb7bl\" (UniqueName: \"kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl\") pod \"maas-api-e2e-derive-a8061117-94ff85b58-crqzr\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.478877 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.478853    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:35.617805 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.617783    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:14:35.620352 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:14:35.620303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab9d7b2_aa34_4594_bbf8_0ab0d779d375.slice/crio-d16870b81e051c2bc611c83a65d148906588ac0fa6a86be6ea3eba65d33cfc94 WatchSource:0}: Error finding container d16870b81e051c2bc611c83a65d148906588ac0fa6a86be6ea3eba65d33cfc94: Status 404 returned error can't find the container with id d16870b81e051c2bc611c83a65d148906588ac0fa6a86be6ea3eba65d33cfc94
Jul 17 19:14:35.833677 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:35.833645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" event={"ID":"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375","Type":"ContainerStarted","Data":"d16870b81e051c2bc611c83a65d148906588ac0fa6a86be6ea3eba65d33cfc94"}
Jul 17 19:14:36.840161 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:36.840126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" event={"ID":"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375","Type":"ContainerStarted","Data":"afaace65d4a3495c258b5d62888255f997f524dcd12bf72855cd8ea9334c5963"}
Jul 17 19:14:36.840564 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:36.840239    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:36.861799 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:36.861744    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" podStartSLOduration=1.466297682 podStartE2EDuration="1.861729022s" totalImagesPullingTime="395.43134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:14:36.860033215 +0000 UTC m=+1802.622463474" watchObservedRunningTime="2026-07-17 19:14:36.861729022 +0000 UTC m=+1802.624159298"
Jul 17 19:14:42.850120 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:42.850086    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:14:59.887927 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:59.887892    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"]
Jul 17 19:14:59.892217 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:59.892200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:14:59.894889 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:59.894866    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:14:59.900854 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:59.900821    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"]
Jul 17 19:14:59.961644 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:14:59.961614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8061117-bbqxd\" (UID: \"8c0f1481-1a50-47ed-8c83-81b984a35bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:00.062899 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.062864    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8061117-bbqxd\" (UID: \"8c0f1481-1a50-47ed-8c83-81b984a35bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:00.063563 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.063537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8061117-bbqxd\" (UID: \"8c0f1481-1a50-47ed-8c83-81b984a35bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:00.132448 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.132412    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"]
Jul 17 19:15:00.135974 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.135952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"
Jul 17 19:15:00.144768 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.144745    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"]
Jul 17 19:15:00.146377 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.146358    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"
Jul 17 19:15:00.204017 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.203987    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:00.296060 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.296035    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"]
Jul 17 19:15:00.298390 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:15:00.298360    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa160c79_7237_4904_9b0b_6ea223a6e848.slice/crio-6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba WatchSource:0}: Error finding container 6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba: Status 404 returned error can't find the container with id 6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba
Jul 17 19:15:00.357406 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.357385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"]
Jul 17 19:15:00.359072 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:15:00.359049    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0f1481_1a50_47ed_8c83_81b984a35bec.slice/crio-10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889 WatchSource:0}: Error finding container 10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889: Status 404 returned error can't find the container with id 10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889
Jul 17 19:15:00.949418 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.949381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn" event={"ID":"fa160c79-7237-4904-9b0b-6ea223a6e848","Type":"ContainerStarted","Data":"e87aa8e715010c0aa8b9dcd866d6274feed9ac5fd9b77fc4eb5bf7385621eed3"}
Jul 17 19:15:00.949830 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.949436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn" event={"ID":"fa160c79-7237-4904-9b0b-6ea223a6e848","Type":"ContainerStarted","Data":"6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba"}
Jul 17 19:15:00.950847 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.950818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd" event={"ID":"8c0f1481-1a50-47ed-8c83-81b984a35bec","Type":"ContainerStarted","Data":"e83d6895e710c9b2fcc0b71cafd56b97d832055fa870f3fe9b1f6a4975345c20"}
Jul 17 19:15:00.950946 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.950861    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd" event={"ID":"8c0f1481-1a50-47ed-8c83-81b984a35bec","Type":"ContainerStarted","Data":"10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889"}
Jul 17 19:15:00.965911 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.965829    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn" podStartSLOduration=0.965814533 podStartE2EDuration="965.814533ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:00.965482191 +0000 UTC m=+1826.727912451" watchObservedRunningTime="2026-07-17 19:15:00.965814533 +0000 UTC m=+1826.728244792"
Jul 17 19:15:00.981905 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:00.981860    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd" podStartSLOduration=1.9818463720000001 podStartE2EDuration="1.981846372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:15:00.980898377 +0000 UTC m=+1826.743328637" watchObservedRunningTime="2026-07-17 19:15:00.981846372 +0000 UTC m=+1826.744276631"
Jul 17 19:15:01.956295 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:01.956262    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa160c79-7237-4904-9b0b-6ea223a6e848" containerID="e87aa8e715010c0aa8b9dcd866d6274feed9ac5fd9b77fc4eb5bf7385621eed3" exitCode=0
Jul 17 19:15:01.956763 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:01.956352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn" event={"ID":"fa160c79-7237-4904-9b0b-6ea223a6e848","Type":"ContainerDied","Data":"e87aa8e715010c0aa8b9dcd866d6274feed9ac5fd9b77fc4eb5bf7385621eed3"}
Jul 17 19:15:01.957794 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:01.957765    2558 generic.go:356] "Generic (PLEG): container finished" podID="8c0f1481-1a50-47ed-8c83-81b984a35bec" containerID="e83d6895e710c9b2fcc0b71cafd56b97d832055fa870f3fe9b1f6a4975345c20" exitCode=0
Jul 17 19:15:01.957907 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:01.957790    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd" event={"ID":"8c0f1481-1a50-47ed-8c83-81b984a35bec","Type":"ContainerDied","Data":"e83d6895e710c9b2fcc0b71cafd56b97d832055fa870f3fe9b1f6a4975345c20"}
Jul 17 19:15:03.114865 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.114839    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:03.132500 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.132475    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"
Jul 17 19:15:03.190133 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.190099    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca\") pod \"8c0f1481-1a50-47ed-8c83-81b984a35bec\" (UID: \"8c0f1481-1a50-47ed-8c83-81b984a35bec\") "
Jul 17 19:15:03.190493 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.190471    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c0f1481-1a50-47ed-8c83-81b984a35bec" (UID: "8c0f1481-1a50-47ed-8c83-81b984a35bec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:03.291622 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.291543    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0f1481-1a50-47ed-8c83-81b984a35bec-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:15:03.968152 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.968118    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"
Jul 17 19:15:03.968152 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.968149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn" event={"ID":"fa160c79-7237-4904-9b0b-6ea223a6e848","Type":"ContainerDied","Data":"6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba"}
Jul 17 19:15:03.968445 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.968198    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6faade24f7d0a90b141dc1668b1cbf437f9df5edf918c3103e50158bbc1410ba"
Jul 17 19:15:03.969384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.969366    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"
Jul 17 19:15:03.969384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.969373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd" event={"ID":"8c0f1481-1a50-47ed-8c83-81b984a35bec","Type":"ContainerDied","Data":"10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889"}
Jul 17 19:15:03.969515 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:03.969410    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10a03c98384ea341a1827b21c49fd9265d3a2d41713487856521d342398d9889"
Jul 17 19:15:10.783904 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:10.783872    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:15:10.784397 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:10.784083    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" podUID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" containerName="maas-api" containerID="cri-o://afaace65d4a3495c258b5d62888255f997f524dcd12bf72855cd8ea9334c5963" gracePeriod=30
Jul 17 19:15:10.832163 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:10.832133    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"]
Jul 17 19:15:10.835919 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:10.835896    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a8061117-29738595-45lmn"]
Jul 17 19:15:11.004407 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.004370    2558 generic.go:356] "Generic (PLEG): container finished" podID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" containerID="afaace65d4a3495c258b5d62888255f997f524dcd12bf72855cd8ea9334c5963" exitCode=0
Jul 17 19:15:11.004856 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.004442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" event={"ID":"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375","Type":"ContainerDied","Data":"afaace65d4a3495c258b5d62888255f997f524dcd12bf72855cd8ea9334c5963"}
Jul 17 19:15:11.033197 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.033170    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:15:11.165082 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.165051    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb7bl\" (UniqueName: \"kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl\") pod \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") "
Jul 17 19:15:11.165262 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.165126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls\") pod \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") "
Jul 17 19:15:11.165262 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.165178    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca\") pod \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\" (UID: \"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375\") "
Jul 17 19:15:11.165614 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.165590    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" (UID: "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:15:11.167371 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.167347    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl" (OuterVolumeSpecName: "kube-api-access-xb7bl") pod "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" (UID: "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375"). InnerVolumeSpecName "kube-api-access-xb7bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:15:11.167450 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.167380    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" (UID: "8ab9d7b2-aa34-4594-bbf8-0ab0d779d375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:15:11.266533 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.266501    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:15:11.266533 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.266528    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:15:11.266533 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:11.266539    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb7bl\" (UniqueName: \"kubernetes.io/projected/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375-kube-api-access-xb7bl\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:15:12.010574 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.010542    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr" event={"ID":"8ab9d7b2-aa34-4594-bbf8-0ab0d779d375","Type":"ContainerDied","Data":"d16870b81e051c2bc611c83a65d148906588ac0fa6a86be6ea3eba65d33cfc94"}
Jul 17 19:15:12.010574 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.010566    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"
Jul 17 19:15:12.011056 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.010596    2558 scope.go:117] "RemoveContainer" containerID="afaace65d4a3495c258b5d62888255f997f524dcd12bf72855cd8ea9334c5963"
Jul 17 19:15:12.032647 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.032616    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:15:12.038682 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.038656    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a8061117-94ff85b58-crqzr"]
Jul 17 19:15:12.773295 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.773260    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" path="/var/lib/kubelet/pods/8ab9d7b2-aa34-4594-bbf8-0ab0d779d375/volumes"
Jul 17 19:15:12.773597 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:12.773582    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa160c79-7237-4904-9b0b-6ea223a6e848" path="/var/lib/kubelet/pods/fa160c79-7237-4904-9b0b-6ea223a6e848/volumes"
Jul 17 19:15:14.977819 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:14.977787    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"]
Jul 17 19:15:14.980734 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:14.980709    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8061117-bbqxd"]
Jul 17 19:15:16.777580 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:15:16.774819    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0f1481-1a50-47ed-8c83-81b984a35bec" path="/var/lib/kubelet/pods/8c0f1481-1a50-47ed-8c83-81b984a35bec/volumes"
Jul 17 19:16:59.484289 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484249    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484602    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa160c79-7237-4904-9b0b-6ea223a6e848" containerName="cleanup"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484614    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa160c79-7237-4904-9b0b-6ea223a6e848" containerName="cleanup"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484623    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0f1481-1a50-47ed-8c83-81b984a35bec" containerName="revoke-keys"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484629    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0f1481-1a50-47ed-8c83-81b984a35bec" containerName="revoke-keys"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484639    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" containerName="maas-api"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484644    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" containerName="maas-api"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484699    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0f1481-1a50-47ed-8c83-81b984a35bec" containerName="revoke-keys"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484706    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa160c79-7237-4904-9b0b-6ea223a6e848" containerName="cleanup"
Jul 17 19:16:59.484797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.484713    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9d7b2-aa34-4594-bbf8-0ab0d779d375" containerName="maas-api"
Jul 17 19:16:59.487872 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.487855    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.492777 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.492740    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1db770c3-serving-cert\""
Jul 17 19:16:59.492914 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.492847    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:16:59.493293 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.493274    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:16:59.493415 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.493347    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:16:59.510008 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.509981    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:16:59.534497 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.534462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.534666 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.534518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.534666 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.534600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrl5\" (UniqueName: \"kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.635883 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.635838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.636074 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.635913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.636074 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.635957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrl5\" (UniqueName: \"kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.636207 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:16:59.636079    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-serving-cert: secret "maas-api-e2e-mt-1db770c3-serving-cert" not found
Jul 17 19:16:59.636207 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:16:59.636184    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls podName:a8bc823d-73d1-43cd-acce-4be9193c9cad nodeName:}" failed. No retries permitted until 2026-07-17 19:17:00.136148226 +0000 UTC m=+1945.898578464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls") pod "maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" (UID: "a8bc823d-73d1-43cd-acce-4be9193c9cad") : secret "maas-api-e2e-mt-1db770c3-serving-cert" not found
Jul 17 19:16:59.636613 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.636594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:16:59.655120 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:16:59.655088    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrl5\" (UniqueName: \"kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:00.139795 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:00.139756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:00.142423 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:00.142392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") pod \"maas-api-e2e-mt-1db770c3-96d854bcd-p9prv\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:00.398990 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:00.398888    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:00.554314 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:00.554290    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:17:00.556938 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:17:00.556910    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc823d_73d1_43cd_acce_4be9193c9cad.slice/crio-da60e29bab5ff214389496fd8573d78452b95078348133408c710fbf584fc082 WatchSource:0}: Error finding container da60e29bab5ff214389496fd8573d78452b95078348133408c710fbf584fc082: Status 404 returned error can't find the container with id da60e29bab5ff214389496fd8573d78452b95078348133408c710fbf584fc082
Jul 17 19:17:01.539105 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:01.539020    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" event={"ID":"a8bc823d-73d1-43cd-acce-4be9193c9cad","Type":"ContainerStarted","Data":"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"}
Jul 17 19:17:01.539105 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:01.539072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" event={"ID":"a8bc823d-73d1-43cd-acce-4be9193c9cad","Type":"ContainerStarted","Data":"da60e29bab5ff214389496fd8573d78452b95078348133408c710fbf584fc082"}
Jul 17 19:17:01.539292 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:01.539109    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:01.565305 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:01.565251    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" podStartSLOduration=2.007486065 podStartE2EDuration="2.565233966s" totalImagesPullingTime="557.747901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:17:01.563910382 +0000 UTC m=+1947.326340641" watchObservedRunningTime="2026-07-17 19:17:01.565233966 +0000 UTC m=+1947.327664219"
Jul 17 19:17:07.549716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:07.549688    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:11.691689 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.691656    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"]
Jul 17 19:17:11.694955 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.694926    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:11.697871 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.697828    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:17:11.705928 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.705906    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"]
Jul 17 19:17:11.752814 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.752780    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1db770c3-hfprz\" (UID: \"f97eee58-8abe-49a1-ba15-80facd15c882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:11.853861 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.853826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1db770c3-hfprz\" (UID: \"f97eee58-8abe-49a1-ba15-80facd15c882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:11.854534 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:11.854512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1db770c3-hfprz\" (UID: \"f97eee58-8abe-49a1-ba15-80facd15c882\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:12.007600 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:12.007521    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:12.356901 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:12.356878    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"]
Jul 17 19:17:12.359227 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:17:12.359197    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97eee58_8abe_49a1_ba15_80facd15c882.slice/crio-c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7 WatchSource:0}: Error finding container c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7: Status 404 returned error can't find the container with id c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7
Jul 17 19:17:12.594713 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:12.594681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz" event={"ID":"f97eee58-8abe-49a1-ba15-80facd15c882","Type":"ContainerStarted","Data":"c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7"}
Jul 17 19:17:13.601232 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:13.601197    2558 generic.go:356] "Generic (PLEG): container finished" podID="f97eee58-8abe-49a1-ba15-80facd15c882" containerID="f5abfccdfd1170cbf3c04240498fae3a8b475c3c921d0272bdd394226552bf4f" exitCode=0
Jul 17 19:17:13.601651 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:13.601266    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz" event={"ID":"f97eee58-8abe-49a1-ba15-80facd15c882","Type":"ContainerDied","Data":"f5abfccdfd1170cbf3c04240498fae3a8b475c3c921d0272bdd394226552bf4f"}
Jul 17 19:17:14.736926 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:14.736893    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:14.782113 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:14.782075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca\") pod \"f97eee58-8abe-49a1-ba15-80facd15c882\" (UID: \"f97eee58-8abe-49a1-ba15-80facd15c882\") "
Jul 17 19:17:14.782396 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:14.782375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f97eee58-8abe-49a1-ba15-80facd15c882" (UID: "f97eee58-8abe-49a1-ba15-80facd15c882"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:14.883817 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:14.883730    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97eee58-8abe-49a1-ba15-80facd15c882-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:17:15.615827 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:15.615787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz" event={"ID":"f97eee58-8abe-49a1-ba15-80facd15c882","Type":"ContainerDied","Data":"c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7"}
Jul 17 19:17:15.615827 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:15.615815    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"
Jul 17 19:17:15.616053 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:15.615874    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64cba7a2c45e0e4460f0f6b0784ce68e1a798dfbd7475c14ecfdc0796f66bd7"
Jul 17 19:17:22.313307 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.313271    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:17:22.313898 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.313864    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerName="maas-api" containerID="cri-o://fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c" gracePeriod=30
Jul 17 19:17:22.568886 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.568819    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:22.651844 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.651808    2558 generic.go:356] "Generic (PLEG): container finished" podID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerID="fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c" exitCode=0
Jul 17 19:17:22.652044 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.651866    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"
Jul 17 19:17:22.652044 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.651900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" event={"ID":"a8bc823d-73d1-43cd-acce-4be9193c9cad","Type":"ContainerDied","Data":"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"}
Jul 17 19:17:22.652044 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.651975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" event={"ID":"a8bc823d-73d1-43cd-acce-4be9193c9cad","Type":"ContainerDied","Data":"da60e29bab5ff214389496fd8573d78452b95078348133408c710fbf584fc082"}
Jul 17 19:17:22.652044 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.651997    2558 scope.go:117] "RemoveContainer" containerID="fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"
Jul 17 19:17:22.658172 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.658149    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca\") pod \"a8bc823d-73d1-43cd-acce-4be9193c9cad\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") "
Jul 17 19:17:22.658301 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.658211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcrl5\" (UniqueName: \"kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5\") pod \"a8bc823d-73d1-43cd-acce-4be9193c9cad\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") "
Jul 17 19:17:22.658301 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.658279    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") pod \"a8bc823d-73d1-43cd-acce-4be9193c9cad\" (UID: \"a8bc823d-73d1-43cd-acce-4be9193c9cad\") "
Jul 17 19:17:22.658580 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.658557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8bc823d-73d1-43cd-acce-4be9193c9cad" (UID: "a8bc823d-73d1-43cd-acce-4be9193c9cad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:17:22.658667 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.658654    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc823d-73d1-43cd-acce-4be9193c9cad-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:17:22.660301 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.660276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5" (OuterVolumeSpecName: "kube-api-access-kcrl5") pod "a8bc823d-73d1-43cd-acce-4be9193c9cad" (UID: "a8bc823d-73d1-43cd-acce-4be9193c9cad"). InnerVolumeSpecName "kube-api-access-kcrl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:17:22.660711 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.660695    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8bc823d-73d1-43cd-acce-4be9193c9cad" (UID: "a8bc823d-73d1-43cd-acce-4be9193c9cad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:17:22.661686 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.661671    2558 scope.go:117] "RemoveContainer" containerID="fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"
Jul 17 19:17:22.661998 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:17:22.661971    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c\": container with ID starting with fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c not found: ID does not exist" containerID="fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"
Jul 17 19:17:22.662050 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.662001    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c"} err="failed to get container status \"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c\": rpc error: code = NotFound desc = could not find container \"fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c\": container with ID starting with fd7a4ba8714823b561a578760114807338b3b2bd925c644624ac15993a34e95c not found: ID does not exist"
Jul 17 19:17:22.759431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.759391    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrl5\" (UniqueName: \"kubernetes.io/projected/a8bc823d-73d1-43cd-acce-4be9193c9cad-kube-api-access-kcrl5\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:17:22.759431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.759426    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8bc823d-73d1-43cd-acce-4be9193c9cad-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:17:22.968836 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.968749    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:17:22.972560 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:22.972533    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv"]
Jul 17 19:17:24.773974 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:24.773940    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" path="/var/lib/kubelet/pods/a8bc823d-73d1-43cd-acce-4be9193c9cad/volumes"
Jul 17 19:17:25.545248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:25.545202    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1db770c3-96d854bcd-p9prv" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.40:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 17 19:17:26.773791 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:26.773760    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"]
Jul 17 19:17:26.778820 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:26.778797    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1db770c3-hfprz"]
Jul 17 19:17:28.773423 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:28.773388    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97eee58-8abe-49a1-ba15-80facd15c882" path="/var/lib/kubelet/pods/f97eee58-8abe-49a1-ba15-80facd15c882/volumes"
Jul 17 19:17:56.721337 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721287    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721638    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerName="maas-api"
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721651    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerName="maas-api"
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721659    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97eee58-8abe-49a1-ba15-80facd15c882" containerName="revoke-keys"
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721664    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97eee58-8abe-49a1-ba15-80facd15c882" containerName="revoke-keys"
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721719    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97eee58-8abe-49a1-ba15-80facd15c882" containerName="revoke-keys"
Jul 17 19:17:56.721797 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.721732    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8bc823d-73d1-43cd-acce-4be9193c9cad" containerName="maas-api"
Jul 17 19:17:56.724781 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.724761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.727456 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.727436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dc79a049-serving-cert\""
Jul 17 19:17:56.727677 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.727663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:17:56.727900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.727885    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:17:56.729738 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.729718    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:17:56.741494 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.741469    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:17:56.752032 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.752000    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.752209 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.752145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwc55\" (UniqueName: \"kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.752209 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.752179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.853550 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.853515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwc55\" (UniqueName: \"kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.853550 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.853555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.853770 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.853616    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.854260 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.854238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.856227 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.856203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:56.865347 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:56.865309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwc55\" (UniqueName: \"kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55\") pod \"maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:57.035067 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.034973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:57.193271 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.193246    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:17:57.196311 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:17:57.196281    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d8226e_e4ce_4210_b56c_764ac283a452.slice/crio-3d0ffcd80c5cb87f7f52067e83e273ae2d2b7d628bdd83cb1a48dc4b3f08d92e WatchSource:0}: Error finding container 3d0ffcd80c5cb87f7f52067e83e273ae2d2b7d628bdd83cb1a48dc4b3f08d92e: Status 404 returned error can't find the container with id 3d0ffcd80c5cb87f7f52067e83e273ae2d2b7d628bdd83cb1a48dc4b3f08d92e
Jul 17 19:17:57.827477 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.827440    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" event={"ID":"86d8226e-e4ce-4210-b56c-764ac283a452","Type":"ContainerStarted","Data":"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"}
Jul 17 19:17:57.827862 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.827496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" event={"ID":"86d8226e-e4ce-4210-b56c-764ac283a452","Type":"ContainerStarted","Data":"3d0ffcd80c5cb87f7f52067e83e273ae2d2b7d628bdd83cb1a48dc4b3f08d92e"}
Jul 17 19:17:57.827862 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.827519    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:17:57.853876 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:17:57.853825    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" podStartSLOduration=1.516008851 podStartE2EDuration="1.853808826s" totalImagesPullingTime="337.799975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:17:57.851551486 +0000 UTC m=+2003.613981745" watchObservedRunningTime="2026-07-17 19:17:57.853808826 +0000 UTC m=+2003.616239086"
Jul 17 19:18:03.839559 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:18:03.839529    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:19:00.231908 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.231871    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"]
Jul 17 19:19:00.235420 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.235401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:00.238938 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.238921    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:19:00.256560 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.256529    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"]
Jul 17 19:19:00.347106 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.347068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2\" (UID: \"10005efe-1625-4297-8b59-d0fd2ccf2f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:00.448224 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.448191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2\" (UID: \"10005efe-1625-4297-8b59-d0fd2ccf2f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:00.448856 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.448835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2\" (UID: \"10005efe-1625-4297-8b59-d0fd2ccf2f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:00.545042 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.544942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:00.688112 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:00.688075    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"]
Jul 17 19:19:00.690518 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:19:00.690478    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10005efe_1625_4297_8b59_d0fd2ccf2f5f.slice/crio-2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1 WatchSource:0}: Error finding container 2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1: Status 404 returned error can't find the container with id 2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1
Jul 17 19:19:01.142820 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:01.142779    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2" event={"ID":"10005efe-1625-4297-8b59-d0fd2ccf2f5f","Type":"ContainerStarted","Data":"1e12bfcee417f3eaf89e8a3387f184c70267fbba9b084bf9cc8bbfdb9022cb63"}
Jul 17 19:19:01.143028 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:01.142861    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2" event={"ID":"10005efe-1625-4297-8b59-d0fd2ccf2f5f","Type":"ContainerStarted","Data":"2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1"}
Jul 17 19:19:01.164965 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:01.164855    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2" podStartSLOduration=1.164837925 podStartE2EDuration="1.164837925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:19:01.162134114 +0000 UTC m=+2066.924564374" watchObservedRunningTime="2026-07-17 19:19:01.164837925 +0000 UTC m=+2066.927268184"
Jul 17 19:19:02.149083 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:02.149046    2558 generic.go:356] "Generic (PLEG): container finished" podID="10005efe-1625-4297-8b59-d0fd2ccf2f5f" containerID="1e12bfcee417f3eaf89e8a3387f184c70267fbba9b084bf9cc8bbfdb9022cb63" exitCode=0
Jul 17 19:19:02.149531 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:02.149143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2" event={"ID":"10005efe-1625-4297-8b59-d0fd2ccf2f5f","Type":"ContainerDied","Data":"1e12bfcee417f3eaf89e8a3387f184c70267fbba9b084bf9cc8bbfdb9022cb63"}
Jul 17 19:19:03.287559 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:03.287537    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:03.474787 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:03.474699    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca\") pod \"10005efe-1625-4297-8b59-d0fd2ccf2f5f\" (UID: \"10005efe-1625-4297-8b59-d0fd2ccf2f5f\") "
Jul 17 19:19:03.475081 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:03.475057    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10005efe-1625-4297-8b59-d0fd2ccf2f5f" (UID: "10005efe-1625-4297-8b59-d0fd2ccf2f5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:03.576311 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:03.576270    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10005efe-1625-4297-8b59-d0fd2ccf2f5f-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:19:04.159472 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:04.159444    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"
Jul 17 19:19:04.159661 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:04.159484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2" event={"ID":"10005efe-1625-4297-8b59-d0fd2ccf2f5f","Type":"ContainerDied","Data":"2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1"}
Jul 17 19:19:04.159661 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:04.159542    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0234e234e3a3a33efe378ace08c63185f168982d0765f8cdc014db074e23c1"
Jul 17 19:19:15.870123 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:15.870086    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:19:15.870923 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:15.870422    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" podUID="86d8226e-e4ce-4210-b56c-764ac283a452" containerName="maas-api" containerID="cri-o://2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e" gracePeriod=30
Jul 17 19:19:16.119034 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.119007    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:19:16.194854 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.194763    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls\") pod \"86d8226e-e4ce-4210-b56c-764ac283a452\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") "
Jul 17 19:19:16.195034 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.194866    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwc55\" (UniqueName: \"kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55\") pod \"86d8226e-e4ce-4210-b56c-764ac283a452\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") "
Jul 17 19:19:16.195034 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.194922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca\") pod \"86d8226e-e4ce-4210-b56c-764ac283a452\" (UID: \"86d8226e-e4ce-4210-b56c-764ac283a452\") "
Jul 17 19:19:16.195292 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.195260    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86d8226e-e4ce-4210-b56c-764ac283a452" (UID: "86d8226e-e4ce-4210-b56c-764ac283a452"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:19:16.196962 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.196934    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86d8226e-e4ce-4210-b56c-764ac283a452" (UID: "86d8226e-e4ce-4210-b56c-764ac283a452"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:19:16.197097 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.197051    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55" (OuterVolumeSpecName: "kube-api-access-nwc55") pod "86d8226e-e4ce-4210-b56c-764ac283a452" (UID: "86d8226e-e4ce-4210-b56c-764ac283a452"). InnerVolumeSpecName "kube-api-access-nwc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:19:16.218564 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.218537    2558 generic.go:356] "Generic (PLEG): container finished" podID="86d8226e-e4ce-4210-b56c-764ac283a452" containerID="2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e" exitCode=0
Jul 17 19:19:16.218704 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.218582    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"
Jul 17 19:19:16.218704 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.218620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" event={"ID":"86d8226e-e4ce-4210-b56c-764ac283a452","Type":"ContainerDied","Data":"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"}
Jul 17 19:19:16.218704 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.218682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw" event={"ID":"86d8226e-e4ce-4210-b56c-764ac283a452","Type":"ContainerDied","Data":"3d0ffcd80c5cb87f7f52067e83e273ae2d2b7d628bdd83cb1a48dc4b3f08d92e"}
Jul 17 19:19:16.218873 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.218715    2558 scope.go:117] "RemoveContainer" containerID="2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"
Jul 17 19:19:16.227991 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.227975    2558 scope.go:117] "RemoveContainer" containerID="2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"
Jul 17 19:19:16.228279 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:19:16.228262    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e\": container with ID starting with 2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e not found: ID does not exist" containerID="2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"
Jul 17 19:19:16.228351 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.228289    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e"} err="failed to get container status \"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e\": rpc error: code = NotFound desc = could not find container \"2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e\": container with ID starting with 2b908b888db3cc2a861b262da796df370e20e9125c037a6ce9c69f8ae5a9ad0e not found: ID does not exist"
Jul 17 19:19:16.243776 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.243751    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:19:16.249522 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.249499    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dc79a049-6f89dd6c46-g2cxw"]
Jul 17 19:19:16.295868 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.295831    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d8226e-e4ce-4210-b56c-764ac283a452-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.295868 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.295862    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwc55\" (UniqueName: \"kubernetes.io/projected/86d8226e-e4ce-4210-b56c-764ac283a452-kube-api-access-nwc55\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.296065 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.295878    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d8226e-e4ce-4210-b56c-764ac283a452-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:19:16.772638 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:16.772610    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d8226e-e4ce-4210-b56c-764ac283a452" path="/var/lib/kubelet/pods/86d8226e-e4ce-4210-b56c-764ac283a452/volumes"
Jul 17 19:19:20.378762 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:20.378726    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"]
Jul 17 19:19:20.384357 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:20.384334    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-dc79a049-pvvw2"]
Jul 17 19:19:20.772769 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:20.772691    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10005efe-1625-4297-8b59-d0fd2ccf2f5f" path="/var/lib/kubelet/pods/10005efe-1625-4297-8b59-d0fd2ccf2f5f/volumes"
Jul 17 19:19:34.841094 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:34.841064    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:19:34.843101 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:34.843083    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:19:34.912951 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:34.912923    2558 scope.go:117] "RemoveContainer" containerID="2b5470b9c940b9269c89edd9a1fd9652e32856de2eaa5ae27754dad7031193b9"
Jul 17 19:19:34.921369 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:19:34.921349    2558 scope.go:117] "RemoveContainer" containerID="37d14b7dfb2fcfe4c7c22f3e527acc44ffa83aa259538f13d74688340f61ea34"
Jul 17 19:20:34.950368 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:20:34.950275    2558 scope.go:117] "RemoveContainer" containerID="3aee3d838a827bfcd4393d4ff728f0c9ad2d7e80d1c071a1b67d6b6ab76c23c2"
Jul 17 19:21:10.694725 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.694690    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695069    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d8226e-e4ce-4210-b56c-764ac283a452" containerName="maas-api"
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695082    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d8226e-e4ce-4210-b56c-764ac283a452" containerName="maas-api"
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695100    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10005efe-1625-4297-8b59-d0fd2ccf2f5f" containerName="revoke-keys"
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695106    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="10005efe-1625-4297-8b59-d0fd2ccf2f5f" containerName="revoke-keys"
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695162    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d8226e-e4ce-4210-b56c-764ac283a452" containerName="maas-api"
Jul 17 19:21:10.695248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.695172    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="10005efe-1625-4297-8b59-d0fd2ccf2f5f" containerName="revoke-keys"
Jul 17 19:21:10.698528 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.698503    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.700909 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.700884    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-26d0e5-serving-cert\""
Jul 17 19:21:10.701051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.700913    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:21:10.701051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.700942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:21:10.701785 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.701767    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:21:10.708837 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.708808    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:21:10.839475 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.839441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.839673 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.839519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwpqk\" (UniqueName: \"kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.839673 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.839586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.940906 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.940865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwpqk\" (UniqueName: \"kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.941126 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.940971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.941126 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.941005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.941728 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.941705    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.943662 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.943645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:10.952090 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:10.952023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwpqk\" (UniqueName: \"kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk\") pod \"maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:11.010616 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.010573    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:11.155495 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.155466    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:21:11.158852 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:21:11.158814    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddaee53_5223_473a_bd66_2f27a2352baf.slice/crio-a341a275c3bac30d32fbc24f3655d11339c5b2966c69ba301e53c04946f10111 WatchSource:0}: Error finding container a341a275c3bac30d32fbc24f3655d11339c5b2966c69ba301e53c04946f10111: Status 404 returned error can't find the container with id a341a275c3bac30d32fbc24f3655d11339c5b2966c69ba301e53c04946f10111
Jul 17 19:21:11.160071 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.160056    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 19:21:11.774633 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.774598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" event={"ID":"1ddaee53-5223-473a-bd66-2f27a2352baf","Type":"ContainerStarted","Data":"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"}
Jul 17 19:21:11.775123 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.774654    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" event={"ID":"1ddaee53-5223-473a-bd66-2f27a2352baf","Type":"ContainerStarted","Data":"a341a275c3bac30d32fbc24f3655d11339c5b2966c69ba301e53c04946f10111"}
Jul 17 19:21:11.775123 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.774696    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:11.794315 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:11.794250    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" podStartSLOduration=1.381455094 podStartE2EDuration="1.794226629s" totalImagesPullingTime="412.771535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:11.792843396 +0000 UTC m=+2197.555273675" watchObservedRunningTime="2026-07-17 19:21:11.794226629 +0000 UTC m=+2197.556656890"
Jul 17 19:21:17.786335 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:17.786300    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:21:18.786516 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.786482    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:21:18.790631 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.790606    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.793201 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.793177    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-912ad7-serving-cert\""
Jul 17 19:21:18.801635 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.801602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.801782 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.801648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.801860 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.801781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hks\" (UniqueName: \"kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.802775 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.802747    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:21:18.902806 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.902773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.902806 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.902808    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.903048 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.902899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hks\" (UniqueName: \"kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.903048 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:21:18.902948    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-serving-cert: secret "maas-api-e2e-inf-b-912ad7-serving-cert" not found
Jul 17 19:21:18.903155 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:21:18.903056    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls podName:1465482a-5094-4b0c-aad5-9f4ff58cf019 nodeName:}" failed. No retries permitted until 2026-07-17 19:21:19.403033971 +0000 UTC m=+2205.165464210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls") pod "maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" (UID: "1465482a-5094-4b0c-aad5-9f4ff58cf019") : secret "maas-api-e2e-inf-b-912ad7-serving-cert" not found
Jul 17 19:21:18.903672 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.903650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:18.915484 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:18.915457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hks\" (UniqueName: \"kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:19.407697 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:19.407661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:19.410248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:19.410223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") pod \"maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:19.703227 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:19.703137    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:19.847867 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:19.847844    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:21:19.849724 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:21:19.849696    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1465482a_5094_4b0c_aad5_9f4ff58cf019.slice/crio-e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c WatchSource:0}: Error finding container e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c: Status 404 returned error can't find the container with id e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c
Jul 17 19:21:20.824051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:20.824006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" event={"ID":"1465482a-5094-4b0c-aad5-9f4ff58cf019","Type":"ContainerStarted","Data":"e4a398839bcd29f432ba170a138c9df43797487effe45fb60bd34ecbba30d034"}
Jul 17 19:21:20.824246 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:20.824076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:20.824246 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:20.824095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" event={"ID":"1465482a-5094-4b0c-aad5-9f4ff58cf019","Type":"ContainerStarted","Data":"e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c"}
Jul 17 19:21:20.853131 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:20.853082    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" podStartSLOduration=2.525350622 podStartE2EDuration="2.853067747s" totalImagesPullingTime="327.717125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:20.851170562 +0000 UTC m=+2206.613600823" watchObservedRunningTime="2026-07-17 19:21:20.853067747 +0000 UTC m=+2206.615498020"
Jul 17 19:21:26.835731 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:26.835699    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:21:34.968192 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:34.968162    2558 scope.go:117] "RemoveContainer" containerID="e87aa8e715010c0aa8b9dcd866d6274feed9ac5fd9b77fc4eb5bf7385621eed3"
Jul 17 19:21:34.976293 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:34.976271    2558 scope.go:117] "RemoveContainer" containerID="e83d6895e710c9b2fcc0b71cafd56b97d832055fa870f3fe9b1f6a4975345c20"
Jul 17 19:21:57.241063 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.241017    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:21:57.245378 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.245352    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.249085 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.249063    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-912ad7\"/\"default-dockercfg-ttpwf\""
Jul 17 19:21:57.250103 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.250085    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-912ad7\"/\"openshift-service-ca.crt\""
Jul 17 19:21:57.250163 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.250097    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-912ad7\"/\"test-model-912ad7-kserve-self-signed-certs\""
Jul 17 19:21:57.250163 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.250118    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-912ad7\"/\"kube-root-ca.crt\""
Jul 17 19:21:57.257787 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.257760    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:21:57.271659 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.271630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.271806 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.271668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.271806 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.271690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79m7\" (UniqueName: \"kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.271806 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.271770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.271998 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.271975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.272107 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.272077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.372851 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.372813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.372851 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.372860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373135 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.372890    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373135 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373135 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373048    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79m7\" (UniqueName: \"kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373135 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373378 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373378 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.373504 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.373484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.375485 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.375462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.375634 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.375613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.388735 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.388702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79m7\" (UniqueName: \"kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7\") pod \"test-model-912ad7-kserve-747589c545-n2tpp\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") " pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.568019 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.567978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:21:57.729860 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.729832    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:21:57.732600 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:21:57.732571    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc467ae68_2af3_4ebf_b26c_0fc65bbdafd3.slice/crio-b62385fd126e8013b4d5a103caaaa5f304916400ba9c0810edb3317fb91e2181 WatchSource:0}: Error finding container b62385fd126e8013b4d5a103caaaa5f304916400ba9c0810edb3317fb91e2181: Status 404 returned error can't find the container with id b62385fd126e8013b4d5a103caaaa5f304916400ba9c0810edb3317fb91e2181
Jul 17 19:21:57.998783 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.998672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" event={"ID":"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3","Type":"ContainerStarted","Data":"8ec08fcf5e7ee73bb4da520a26c282e5d55f8f02aecbcc0f00524e7fdfb83a7a"}
Jul 17 19:21:57.998783 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:57.998729    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" event={"ID":"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3","Type":"ContainerStarted","Data":"b62385fd126e8013b4d5a103caaaa5f304916400ba9c0810edb3317fb91e2181"}
Jul 17 19:21:58.028471 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:21:58.028418    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" podStartSLOduration=1.028402655 podStartE2EDuration="1.028402655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:21:58.027889318 +0000 UTC m=+2243.790319578" watchObservedRunningTime="2026-07-17 19:21:58.028402655 +0000 UTC m=+2243.790832915"
Jul 17 19:22:07.568604 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:07.568561    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:07.569038 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:07.568619    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:07.581891 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:07.581866    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:08.057196 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:08.057170    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:28.018741 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.018705    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:22:28.139227 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.139192    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"]
Jul 17 19:22:28.143141 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.143119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:28.146647 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.146625    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:22:28.154204 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.154179    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"]
Jul 17 19:22:28.267145 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.267097    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2\" (UID: \"9836758c-3190-4339-8453-65ff57db6996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:28.368013 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.367972    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2\" (UID: \"9836758c-3190-4339-8453-65ff57db6996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:28.368630 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.368604    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2\" (UID: \"9836758c-3190-4339-8453-65ff57db6996\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:28.455784 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.455745    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:28.603183 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:28.603159    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"]
Jul 17 19:22:28.606645 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:22:28.606615    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9836758c_3190_4339_8453_65ff57db6996.slice/crio-7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626 WatchSource:0}: Error finding container 7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626: Status 404 returned error can't find the container with id 7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626
Jul 17 19:22:29.147722 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:29.147683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2" event={"ID":"9836758c-3190-4339-8453-65ff57db6996","Type":"ContainerStarted","Data":"69e33243db312a3fab53908d24b1b508b2ab3c81b06d526a430338a5eac9bbcc"}
Jul 17 19:22:29.148132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:29.147746    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2" event={"ID":"9836758c-3190-4339-8453-65ff57db6996","Type":"ContainerStarted","Data":"7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626"}
Jul 17 19:22:29.165144 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:29.165044    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2" podStartSLOduration=1.165027674 podStartE2EDuration="1.165027674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:29.163470886 +0000 UTC m=+2274.925901169" watchObservedRunningTime="2026-07-17 19:22:29.165027674 +0000 UTC m=+2274.927457931"
Jul 17 19:22:30.153373 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:30.153335    2558 generic.go:356] "Generic (PLEG): container finished" podID="9836758c-3190-4339-8453-65ff57db6996" containerID="69e33243db312a3fab53908d24b1b508b2ab3c81b06d526a430338a5eac9bbcc" exitCode=0
Jul 17 19:22:30.153827 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:30.153418    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2" event={"ID":"9836758c-3190-4339-8453-65ff57db6996","Type":"ContainerDied","Data":"69e33243db312a3fab53908d24b1b508b2ab3c81b06d526a430338a5eac9bbcc"}
Jul 17 19:22:31.295299 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:31.295275    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:31.297233 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:31.297214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca\") pod \"9836758c-3190-4339-8453-65ff57db6996\" (UID: \"9836758c-3190-4339-8453-65ff57db6996\") "
Jul 17 19:22:31.297556 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:31.297534    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9836758c-3190-4339-8453-65ff57db6996" (UID: "9836758c-3190-4339-8453-65ff57db6996"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:31.398222 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:31.398181    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9836758c-3190-4339-8453-65ff57db6996-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:32.163691 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:32.163664    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"
Jul 17 19:22:32.163691 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:32.163678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2" event={"ID":"9836758c-3190-4339-8453-65ff57db6996","Type":"ContainerDied","Data":"7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626"}
Jul 17 19:22:32.163915 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:32.163733    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b95383afb51f215652c6344ce36802c45e269c83eec253774460b69c19bf626"
Jul 17 19:22:38.765762 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:38.765727    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:22:38.768186 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:38.766016    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" podUID="1ddaee53-5223-473a-bd66-2f27a2352baf" containerName="maas-api" containerID="cri-o://817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070" gracePeriod=30
Jul 17 19:22:39.015592 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.015564    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:22:39.055564 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.055528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwpqk\" (UniqueName: \"kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk\") pod \"1ddaee53-5223-473a-bd66-2f27a2352baf\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") "
Jul 17 19:22:39.055751 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.055599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca\") pod \"1ddaee53-5223-473a-bd66-2f27a2352baf\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") "
Jul 17 19:22:39.055751 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.055629    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls\") pod \"1ddaee53-5223-473a-bd66-2f27a2352baf\" (UID: \"1ddaee53-5223-473a-bd66-2f27a2352baf\") "
Jul 17 19:22:39.056019 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.055987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ddaee53-5223-473a-bd66-2f27a2352baf" (UID: "1ddaee53-5223-473a-bd66-2f27a2352baf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:39.058014 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.057987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk" (OuterVolumeSpecName: "kube-api-access-bwpqk") pod "1ddaee53-5223-473a-bd66-2f27a2352baf" (UID: "1ddaee53-5223-473a-bd66-2f27a2352baf"). InnerVolumeSpecName "kube-api-access-bwpqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:22:39.058134 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.058094    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ddaee53-5223-473a-bd66-2f27a2352baf" (UID: "1ddaee53-5223-473a-bd66-2f27a2352baf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:22:39.156340 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.156281    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwpqk\" (UniqueName: \"kubernetes.io/projected/1ddaee53-5223-473a-bd66-2f27a2352baf-kube-api-access-bwpqk\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:39.156340 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.156310    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ddaee53-5223-473a-bd66-2f27a2352baf-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:39.156340 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.156346    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ddaee53-5223-473a-bd66-2f27a2352baf-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:39.195861 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.195822    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ddaee53-5223-473a-bd66-2f27a2352baf" containerID="817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070" exitCode=0
Jul 17 19:22:39.196058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.195868    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"
Jul 17 19:22:39.196058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.195892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" event={"ID":"1ddaee53-5223-473a-bd66-2f27a2352baf","Type":"ContainerDied","Data":"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"}
Jul 17 19:22:39.196058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.195949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2" event={"ID":"1ddaee53-5223-473a-bd66-2f27a2352baf","Type":"ContainerDied","Data":"a341a275c3bac30d32fbc24f3655d11339c5b2966c69ba301e53c04946f10111"}
Jul 17 19:22:39.196058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.195974    2558 scope.go:117] "RemoveContainer" containerID="817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"
Jul 17 19:22:39.205735 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.205718    2558 scope.go:117] "RemoveContainer" containerID="817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"
Jul 17 19:22:39.206026 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:22:39.206008    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070\": container with ID starting with 817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070 not found: ID does not exist" containerID="817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"
Jul 17 19:22:39.206077 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.206035    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070"} err="failed to get container status \"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070\": rpc error: code = NotFound desc = could not find container \"817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070\": container with ID starting with 817beb54b938942ccccbdc243528970eedca594c6dccac605ccf268a2e601070 not found: ID does not exist"
Jul 17 19:22:39.220268 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.220238    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:22:39.224685 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:39.224659    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-26d0e5-559c4d95fc-wmlw2"]
Jul 17 19:22:40.773493 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:40.773461    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ddaee53-5223-473a-bd66-2f27a2352baf" path="/var/lib/kubelet/pods/1ddaee53-5223-473a-bd66-2f27a2352baf/volumes"
Jul 17 19:22:43.039621 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.039568    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" podUID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" containerName="main" containerID="cri-o://8ec08fcf5e7ee73bb4da520a26c282e5d55f8f02aecbcc0f00524e7fdfb83a7a" gracePeriod=45
Jul 17 19:22:43.219038 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.219011    2558 generic.go:356] "Generic (PLEG): container finished" podID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" containerID="8ec08fcf5e7ee73bb4da520a26c282e5d55f8f02aecbcc0f00524e7fdfb83a7a" exitCode=0
Jul 17 19:22:43.219212 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.219087    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" event={"ID":"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3","Type":"ContainerDied","Data":"8ec08fcf5e7ee73bb4da520a26c282e5d55f8f02aecbcc0f00524e7fdfb83a7a"}
Jul 17 19:22:43.235564 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.235520    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"]
Jul 17 19:22:43.241384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.239688    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-26d0e5-jt5f2"]
Jul 17 19:22:43.294621 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.294557    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:43.390024 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.389989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390024 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390030    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390132    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390296 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390219    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h79m7\" (UniqueName: \"kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7\") pod \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\" (UID: \"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3\") "
Jul 17 19:22:43.390544 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390416    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:22:43.390544 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home" (OuterVolumeSpecName: "home") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:22:43.390544 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:22:43.390662 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390551    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-home\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:43.390662 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.390578    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-model-cache\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:43.392617 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.392589    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7" (OuterVolumeSpecName: "kube-api-access-h79m7") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "kube-api-access-h79m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:22:43.392731 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.392619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:22:43.392731 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.392623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm" (OuterVolumeSpecName: "dshm") pod "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" (UID: "c467ae68-2af3-4ebf-b26c-0fc65bbdafd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:22:43.492107 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.492067    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-dshm\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:43.492107 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.492106    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tls-certs\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:43.492360 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.492121    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-tmp-dir\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:43.492360 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:43.492138    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79m7\" (UniqueName: \"kubernetes.io/projected/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3-kube-api-access-h79m7\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:44.225471 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.225443    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"
Jul 17 19:22:44.225904 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.225479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp" event={"ID":"c467ae68-2af3-4ebf-b26c-0fc65bbdafd3","Type":"ContainerDied","Data":"b62385fd126e8013b4d5a103caaaa5f304916400ba9c0810edb3317fb91e2181"}
Jul 17 19:22:44.225904 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.225530    2558 scope.go:117] "RemoveContainer" containerID="8ec08fcf5e7ee73bb4da520a26c282e5d55f8f02aecbcc0f00524e7fdfb83a7a"
Jul 17 19:22:44.276552 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.276517    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:22:44.294902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.294868    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-912ad7/test-model-912ad7-kserve-747589c545-n2tpp"]
Jul 17 19:22:44.773460 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.773418    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9836758c-3190-4339-8453-65ff57db6996" path="/var/lib/kubelet/pods/9836758c-3190-4339-8453-65ff57db6996/volumes"
Jul 17 19:22:44.773925 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:44.773904    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" path="/var/lib/kubelet/pods/c467ae68-2af3-4ebf-b26c-0fc65bbdafd3/volumes"
Jul 17 19:22:50.402985 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.402951    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"]
Jul 17 19:22:50.403506 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403486    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ddaee53-5223-473a-bd66-2f27a2352baf" containerName="maas-api"
Jul 17 19:22:50.403583 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403509    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ddaee53-5223-473a-bd66-2f27a2352baf" containerName="maas-api"
Jul 17 19:22:50.403583 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403532    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9836758c-3190-4339-8453-65ff57db6996" containerName="revoke-keys"
Jul 17 19:22:50.403583 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403542    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9836758c-3190-4339-8453-65ff57db6996" containerName="revoke-keys"
Jul 17 19:22:50.403583 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403560    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" containerName="main"
Jul 17 19:22:50.403583 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403569    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" containerName="main"
Jul 17 19:22:50.403826 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403680    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9836758c-3190-4339-8453-65ff57db6996" containerName="revoke-keys"
Jul 17 19:22:50.403826 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403696    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c467ae68-2af3-4ebf-b26c-0fc65bbdafd3" containerName="main"
Jul 17 19:22:50.403826 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.403707    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ddaee53-5223-473a-bd66-2f27a2352baf" containerName="maas-api"
Jul 17 19:22:50.408175 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.408157    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:50.410734 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.410713    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:22:50.414876 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.414852    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"]
Jul 17 19:22:50.454314 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.454285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl\" (UID: \"50f535b2-6677-42fd-94a8-c96ca5d9d51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:50.555459 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.555419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl\" (UID: \"50f535b2-6677-42fd-94a8-c96ca5d9d51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:50.556038 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.556016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl\" (UID: \"50f535b2-6677-42fd-94a8-c96ca5d9d51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:50.719902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.719794    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:50.860932 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:50.860902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"]
Jul 17 19:22:50.863622 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:22:50.863592    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f535b2_6677_42fd_94a8_c96ca5d9d51f.slice/crio-14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a WatchSource:0}: Error finding container 14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a: Status 404 returned error can't find the container with id 14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a
Jul 17 19:22:51.263664 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:51.263627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl" event={"ID":"50f535b2-6677-42fd-94a8-c96ca5d9d51f","Type":"ContainerStarted","Data":"c060a9900b907bfed65427e490c17b8eefd65bee12ace42eabf11cb924fce715"}
Jul 17 19:22:51.263870 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:51.263682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl" event={"ID":"50f535b2-6677-42fd-94a8-c96ca5d9d51f","Type":"ContainerStarted","Data":"14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a"}
Jul 17 19:22:51.280725 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:51.280671    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl" podStartSLOduration=1.280653809 podStartE2EDuration="1.280653809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:51.280050445 +0000 UTC m=+2297.042480705" watchObservedRunningTime="2026-07-17 19:22:51.280653809 +0000 UTC m=+2297.043084074"
Jul 17 19:22:52.269881 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:52.269847    2558 generic.go:356] "Generic (PLEG): container finished" podID="50f535b2-6677-42fd-94a8-c96ca5d9d51f" containerID="c060a9900b907bfed65427e490c17b8eefd65bee12ace42eabf11cb924fce715" exitCode=0
Jul 17 19:22:52.270268 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:52.269917    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl" event={"ID":"50f535b2-6677-42fd-94a8-c96ca5d9d51f","Type":"ContainerDied","Data":"c060a9900b907bfed65427e490c17b8eefd65bee12ace42eabf11cb924fce715"}
Jul 17 19:22:53.404619 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:53.404593    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:53.481049 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:53.481014    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca\") pod \"50f535b2-6677-42fd-94a8-c96ca5d9d51f\" (UID: \"50f535b2-6677-42fd-94a8-c96ca5d9d51f\") "
Jul 17 19:22:53.481452 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:53.481433    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50f535b2-6677-42fd-94a8-c96ca5d9d51f" (UID: "50f535b2-6677-42fd-94a8-c96ca5d9d51f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:22:53.582676 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:53.582646    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f535b2-6677-42fd-94a8-c96ca5d9d51f-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:22:54.280716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:54.280680    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"
Jul 17 19:22:54.280868 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:54.280714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl" event={"ID":"50f535b2-6677-42fd-94a8-c96ca5d9d51f","Type":"ContainerDied","Data":"14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a"}
Jul 17 19:22:54.280868 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:22:54.280765    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14270066c0ddd2948d240a4490e195f219bfaf503af3e6d7d037fac4fd9f935a"
Jul 17 19:23:00.877419 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:00.877386    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:23:00.877982 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:00.877638    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" podUID="1465482a-5094-4b0c-aad5-9f4ff58cf019" containerName="maas-api" containerID="cri-o://e4a398839bcd29f432ba170a138c9df43797487effe45fb60bd34ecbba30d034" gracePeriod=30
Jul 17 19:23:01.314178 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.314147    2558 generic.go:356] "Generic (PLEG): container finished" podID="1465482a-5094-4b0c-aad5-9f4ff58cf019" containerID="e4a398839bcd29f432ba170a138c9df43797487effe45fb60bd34ecbba30d034" exitCode=0
Jul 17 19:23:01.314346 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.314198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" event={"ID":"1465482a-5094-4b0c-aad5-9f4ff58cf019","Type":"ContainerDied","Data":"e4a398839bcd29f432ba170a138c9df43797487effe45fb60bd34ecbba30d034"}
Jul 17 19:23:01.314346 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.314238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw" event={"ID":"1465482a-5094-4b0c-aad5-9f4ff58cf019","Type":"ContainerDied","Data":"e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c"}
Jul 17 19:23:01.314346 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.314255    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7c1c1c1ea3e88b22f7ff906d14679fd772a75c9927cf92ea6400b237440429c"
Jul 17 19:23:01.326384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.326363    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:23:01.450749 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.450669    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") pod \"1465482a-5094-4b0c-aad5-9f4ff58cf019\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") "
Jul 17 19:23:01.450749 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.450718    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca\") pod \"1465482a-5094-4b0c-aad5-9f4ff58cf019\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") "
Jul 17 19:23:01.450749 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.450748    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4hks\" (UniqueName: \"kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks\") pod \"1465482a-5094-4b0c-aad5-9f4ff58cf019\" (UID: \"1465482a-5094-4b0c-aad5-9f4ff58cf019\") "
Jul 17 19:23:01.451068 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.451048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1465482a-5094-4b0c-aad5-9f4ff58cf019" (UID: "1465482a-5094-4b0c-aad5-9f4ff58cf019"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:01.453058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.453039    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1465482a-5094-4b0c-aad5-9f4ff58cf019" (UID: "1465482a-5094-4b0c-aad5-9f4ff58cf019"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:23:01.453119 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.453052    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks" (OuterVolumeSpecName: "kube-api-access-k4hks") pod "1465482a-5094-4b0c-aad5-9f4ff58cf019" (UID: "1465482a-5094-4b0c-aad5-9f4ff58cf019"). InnerVolumeSpecName "kube-api-access-k4hks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:23:01.553809 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.552059    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1465482a-5094-4b0c-aad5-9f4ff58cf019-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:01.553809 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.552101    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1465482a-5094-4b0c-aad5-9f4ff58cf019-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:01.553809 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:01.552177    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hks\" (UniqueName: \"kubernetes.io/projected/1465482a-5094-4b0c-aad5-9f4ff58cf019-kube-api-access-k4hks\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:02.318743 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:02.318706    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"
Jul 17 19:23:02.341019 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:02.340985    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:23:02.342627 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:02.342595    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-912ad7-559857c4fd-jrrsw"]
Jul 17 19:23:02.773338 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:02.773248    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1465482a-5094-4b0c-aad5-9f4ff58cf019" path="/var/lib/kubelet/pods/1465482a-5094-4b0c-aad5-9f4ff58cf019/volumes"
Jul 17 19:23:05.483333 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:05.483299    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"]
Jul 17 19:23:05.485772 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:05.485749    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-912ad7-4pvwl"]
Jul 17 19:23:06.773448 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:06.773414    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f535b2-6677-42fd-94a8-c96ca5d9d51f" path="/var/lib/kubelet/pods/50f535b2-6677-42fd-94a8-c96ca5d9d51f/volumes"
Jul 17 19:23:16.371594 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.371560    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:16.372115 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372097    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1465482a-5094-4b0c-aad5-9f4ff58cf019" containerName="maas-api"
Jul 17 19:23:16.372180 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1465482a-5094-4b0c-aad5-9f4ff58cf019" containerName="maas-api"
Jul 17 19:23:16.372180 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372131    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f535b2-6677-42fd-94a8-c96ca5d9d51f" containerName="revoke-keys"
Jul 17 19:23:16.372180 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372139    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f535b2-6677-42fd-94a8-c96ca5d9d51f" containerName="revoke-keys"
Jul 17 19:23:16.372293 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372260    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1465482a-5094-4b0c-aad5-9f4ff58cf019" containerName="maas-api"
Jul 17 19:23:16.372293 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.372272    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f535b2-6677-42fd-94a8-c96ca5d9d51f" containerName="revoke-keys"
Jul 17 19:23:16.375563 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.375547    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.379016 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.378992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-55c341-serving-cert\""
Jul 17 19:23:16.379138 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.379086    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:23:16.379203 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.379168    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:23:16.379366 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.379348    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:23:16.386627 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.386606    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:16.488485 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.488444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.488660 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.488509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.488660 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.488567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjkj\" (UniqueName: \"kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.589291 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.589255    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.589481 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.589311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjkj\" (UniqueName: \"kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.589481 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.589380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.589481 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:23:16.589437    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-serving-cert: secret "maas-api-e2e-api-a-55c341-serving-cert" not found
Jul 17 19:23:16.589592 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:23:16.589508    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls podName:d2e37b3d-caa4-472f-9be4-a21fd2bff3ea nodeName:}" failed. No retries permitted until 2026-07-17 19:23:17.089492218 +0000 UTC m=+2322.851922456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls") pod "maas-api-e2e-api-a-55c341-565d44cbf-8xvch" (UID: "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea") : secret "maas-api-e2e-api-a-55c341-serving-cert" not found
Jul 17 19:23:16.589976 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.589959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:16.603893 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:16.603868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjkj\" (UniqueName: \"kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:17.095347 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:17.095299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:17.098043 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:17.098012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") pod \"maas-api-e2e-api-a-55c341-565d44cbf-8xvch\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:17.286776 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:17.286738    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:17.459522 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:17.459488    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:17.462157 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:23:17.462130    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e37b3d_caa4_472f_9be4_a21fd2bff3ea.slice/crio-ff1d46d09d92fafe0a87f46d6f6d6b0987e60e1ba39da8b710ff23e2188d1011 WatchSource:0}: Error finding container ff1d46d09d92fafe0a87f46d6f6d6b0987e60e1ba39da8b710ff23e2188d1011: Status 404 returned error can't find the container with id ff1d46d09d92fafe0a87f46d6f6d6b0987e60e1ba39da8b710ff23e2188d1011
Jul 17 19:23:18.401984 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:18.401946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" event={"ID":"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea","Type":"ContainerStarted","Data":"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"}
Jul 17 19:23:18.402193 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:18.402009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" event={"ID":"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea","Type":"ContainerStarted","Data":"ff1d46d09d92fafe0a87f46d6f6d6b0987e60e1ba39da8b710ff23e2188d1011"}
Jul 17 19:23:18.402193 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:18.402054    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:18.426538 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:18.426479    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" podStartSLOduration=1.9413563360000001 podStartE2EDuration="2.426459912s" totalImagesPullingTime="485.103576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:18.424415812 +0000 UTC m=+2324.186846081" watchObservedRunningTime="2026-07-17 19:23:18.426459912 +0000 UTC m=+2324.188890172"
Jul 17 19:23:24.411989 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.411959    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:24.539889 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.539857    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:23:24.543338 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.543295    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.546374 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.546351    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8fe1d7-serving-cert\""
Jul 17 19:23:24.561458 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.561431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.561584 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.561468    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nvq\" (UniqueName: \"kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.561584 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.561530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.567969 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.567940    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:23:24.662069 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.661983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.662233 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.662113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.662233 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.662156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nvq\" (UniqueName: \"kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.662362 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:23:24.662266    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-serving-cert: secret "maas-api-e2e-api-b-8fe1d7-serving-cert" not found
Jul 17 19:23:24.662400 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:23:24.662364    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls podName:a520fe86-fbd5-4b4f-aa79-e84dcafd4a68 nodeName:}" failed. No retries permitted until 2026-07-17 19:23:25.162345899 +0000 UTC m=+2330.924776155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls") pod "maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" (UID: "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68") : secret "maas-api-e2e-api-b-8fe1d7-serving-cert" not found
Jul 17 19:23:24.662775 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.662758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:24.675259 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:24.675230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nvq\" (UniqueName: \"kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:25.167083 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:25.167044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:25.169815 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:25.169791    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") pod \"maas-api-e2e-api-b-8fe1d7-55d586f956-lq522\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:25.454507 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:25.454430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:25.638701 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:25.638674    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:23:25.640068 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:23:25.640045    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda520fe86_fbd5_4b4f_aa79_e84dcafd4a68.slice/crio-e9d64c02b1eaaae28a2d2091f9036a2e2b54d821ee5a3b3f60f2be1eb52b3f62 WatchSource:0}: Error finding container e9d64c02b1eaaae28a2d2091f9036a2e2b54d821ee5a3b3f60f2be1eb52b3f62: Status 404 returned error can't find the container with id e9d64c02b1eaaae28a2d2091f9036a2e2b54d821ee5a3b3f60f2be1eb52b3f62
Jul 17 19:23:26.442518 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:26.442479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" event={"ID":"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68","Type":"ContainerStarted","Data":"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"}
Jul 17 19:23:26.442703 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:26.442547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" event={"ID":"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68","Type":"ContainerStarted","Data":"e9d64c02b1eaaae28a2d2091f9036a2e2b54d821ee5a3b3f60f2be1eb52b3f62"}
Jul 17 19:23:26.442703 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:26.442587    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:26.478458 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:26.478407    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" podStartSLOduration=2.092757217 podStartE2EDuration="2.478389911s" totalImagesPullingTime="385.632694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:23:26.476902809 +0000 UTC m=+2332.239333080" watchObservedRunningTime="2026-07-17 19:23:26.478389911 +0000 UTC m=+2332.240820171"
Jul 17 19:23:32.453179 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:32.453152    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:23:35.005018 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:35.004989    2558 scope.go:117] "RemoveContainer" containerID="f5abfccdfd1170cbf3c04240498fae3a8b475c3c921d0272bdd394226552bf4f"
Jul 17 19:23:37.165659 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.165624    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"]
Jul 17 19:23:37.170384 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.170365    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:37.173434 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.173410    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:23:37.180366 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.180341    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"]
Jul 17 19:23:37.269174 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.269141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55c341-6sh8w\" (UID: \"2670ecc1-6189-41e7-9e24-052c28674909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:37.370256 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.370222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55c341-6sh8w\" (UID: \"2670ecc1-6189-41e7-9e24-052c28674909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:37.370937 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.370918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55c341-6sh8w\" (UID: \"2670ecc1-6189-41e7-9e24-052c28674909\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:37.482905 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.482819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:37.630675 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:37.630642    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"]
Jul 17 19:23:37.633107 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:23:37.633084    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2670ecc1_6189_41e7_9e24_052c28674909.slice/crio-2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91 WatchSource:0}: Error finding container 2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91: Status 404 returned error can't find the container with id 2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91
Jul 17 19:23:38.506309 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:38.506275    2558 generic.go:356] "Generic (PLEG): container finished" podID="2670ecc1-6189-41e7-9e24-052c28674909" containerID="1153724640da6ef932457531745f5fea49436ea919acde5eb4604999c791a2f9" exitCode=0
Jul 17 19:23:38.506731 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:38.506357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w" event={"ID":"2670ecc1-6189-41e7-9e24-052c28674909","Type":"ContainerDied","Data":"1153724640da6ef932457531745f5fea49436ea919acde5eb4604999c791a2f9"}
Jul 17 19:23:38.506731 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:38.506461    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w" event={"ID":"2670ecc1-6189-41e7-9e24-052c28674909","Type":"ContainerStarted","Data":"2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91"}
Jul 17 19:23:39.652761 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:39.652736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:39.689983 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:39.689957    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca\") pod \"2670ecc1-6189-41e7-9e24-052c28674909\" (UID: \"2670ecc1-6189-41e7-9e24-052c28674909\") "
Jul 17 19:23:39.690475 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:39.690447    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2670ecc1-6189-41e7-9e24-052c28674909" (UID: "2670ecc1-6189-41e7-9e24-052c28674909"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:39.790627 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:39.790548    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2670ecc1-6189-41e7-9e24-052c28674909-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:40.517391 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:40.517355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w" event={"ID":"2670ecc1-6189-41e7-9e24-052c28674909","Type":"ContainerDied","Data":"2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91"}
Jul 17 19:23:40.517391 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:40.517376    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"
Jul 17 19:23:40.517633 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:40.517409    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9bcbb1c856aa238ac06a814ea54a4c8f4c6537ceb271f4d50865cebad90f91"
Jul 17 19:23:52.723587 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:52.723552    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:52.724006 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:52.723785    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" podUID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" containerName="maas-api" containerID="cri-o://d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92" gracePeriod=30
Jul 17 19:23:52.974482 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:52.974425    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:53.123617 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.123576    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") pod \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") "
Jul 17 19:23:53.123617 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.123622    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca\") pod \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") "
Jul 17 19:23:53.123870 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.123757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fjkj\" (UniqueName: \"kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj\") pod \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\" (UID: \"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea\") "
Jul 17 19:23:53.124040 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.124006    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" (UID: "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:23:53.125860 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.125833    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" (UID: "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:23:53.126011 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.125989    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj" (OuterVolumeSpecName: "kube-api-access-9fjkj") pod "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" (UID: "d2e37b3d-caa4-472f-9be4-a21fd2bff3ea"). InnerVolumeSpecName "kube-api-access-9fjkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:23:53.224717 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.224620    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjkj\" (UniqueName: \"kubernetes.io/projected/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-kube-api-access-9fjkj\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:53.224717 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.224655    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:53.224717 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.224665    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:23:53.580944 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.580915    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" containerID="d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92" exitCode=0
Jul 17 19:23:53.581106 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.580967    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"
Jul 17 19:23:53.581106 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.581007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" event={"ID":"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea","Type":"ContainerDied","Data":"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"}
Jul 17 19:23:53.581106 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.581061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch" event={"ID":"d2e37b3d-caa4-472f-9be4-a21fd2bff3ea","Type":"ContainerDied","Data":"ff1d46d09d92fafe0a87f46d6f6d6b0987e60e1ba39da8b710ff23e2188d1011"}
Jul 17 19:23:53.581106 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.581086    2558 scope.go:117] "RemoveContainer" containerID="d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"
Jul 17 19:23:53.590294 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.590275    2558 scope.go:117] "RemoveContainer" containerID="d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"
Jul 17 19:23:53.590567 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:23:53.590549    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92\": container with ID starting with d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92 not found: ID does not exist" containerID="d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"
Jul 17 19:23:53.590619 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.590576    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92"} err="failed to get container status \"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92\": rpc error: code = NotFound desc = could not find container \"d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92\": container with ID starting with d7e2d0fbed88ed2aca0f7856d7375ef3472ae877957c83158fa875494513bb92 not found: ID does not exist"
Jul 17 19:23:53.603983 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.603957    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:53.608251 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:53.608227    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55c341-565d44cbf-8xvch"]
Jul 17 19:23:54.772794 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:54.772763    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" path="/var/lib/kubelet/pods/d2e37b3d-caa4-472f-9be4-a21fd2bff3ea/volumes"
Jul 17 19:23:57.354760 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:57.354722    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"]
Jul 17 19:23:57.357272 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:57.357241    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55c341-6sh8w"]
Jul 17 19:23:58.772795 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:23:58.772755    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2670ecc1-6189-41e7-9e24-052c28674909" path="/var/lib/kubelet/pods/2670ecc1-6189-41e7-9e24-052c28674909/volumes"
Jul 17 19:24:04.437392 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437355    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"]
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437744    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" containerName="maas-api"
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437756    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" containerName="maas-api"
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437768    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2670ecc1-6189-41e7-9e24-052c28674909" containerName="revoke-keys"
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437776    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2670ecc1-6189-41e7-9e24-052c28674909" containerName="revoke-keys"
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437832    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2670ecc1-6189-41e7-9e24-052c28674909" containerName="revoke-keys"
Jul 17 19:24:04.437875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.437843    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e37b3d-caa4-472f-9be4-a21fd2bff3ea" containerName="maas-api"
Jul 17 19:24:04.442198 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.442180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:04.445500 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.445463    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:24:04.449955 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.449932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"]
Jul 17 19:24:04.633223 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.633186    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt\" (UID: \"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:04.733898 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.733807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt\" (UID: \"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:04.734502 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.734484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt\" (UID: \"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:04.752732 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.752694    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:04.894127 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:04.894103    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"]
Jul 17 19:24:04.896466 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:24:04.896438    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf8db8_37a3_48f0_ad2c_62fc7fd85b16.slice/crio-a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202 WatchSource:0}: Error finding container a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202: Status 404 returned error can't find the container with id a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202
Jul 17 19:24:05.640928 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:05.640893    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" containerID="29f6780fa5462dbbf8a6ced617945b78a6a16ca5cb3adc8d8fa669c9be916f70" exitCode=0
Jul 17 19:24:05.641374 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:05.640987    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt" event={"ID":"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16","Type":"ContainerDied","Data":"29f6780fa5462dbbf8a6ced617945b78a6a16ca5cb3adc8d8fa669c9be916f70"}
Jul 17 19:24:05.641374 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:05.641048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt" event={"ID":"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16","Type":"ContainerStarted","Data":"a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202"}
Jul 17 19:24:06.778932 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:06.778906    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:06.848661 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:06.848626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca\") pod \"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16\" (UID: \"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16\") "
Jul 17 19:24:06.849057 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:06.849033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" (UID: "f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:06.949360 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:06.949246    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:24:07.653124 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:07.653091    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"
Jul 17 19:24:07.653347 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:07.653129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt" event={"ID":"f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16","Type":"ContainerDied","Data":"a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202"}
Jul 17 19:24:07.653347 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:07.653185    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62a3a9c2f7e79178ebaa5db5dcd57a9edc1a6ba67babd84cb991ba907cb6202"
Jul 17 19:24:20.080762 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.080727    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:24:20.081142 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.080946    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" podUID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" containerName="maas-api" containerID="cri-o://db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1" gracePeriod=30
Jul 17 19:24:20.334662 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.334608    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:24:20.362128 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.362103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47nvq\" (UniqueName: \"kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq\") pod \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") "
Jul 17 19:24:20.362248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.362147    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") pod \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") "
Jul 17 19:24:20.362248 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.362166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca\") pod \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\" (UID: \"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68\") "
Jul 17 19:24:20.362531 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.362511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" (UID: "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:24:20.364354 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.364335    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq" (OuterVolumeSpecName: "kube-api-access-47nvq") pod "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" (UID: "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68"). InnerVolumeSpecName "kube-api-access-47nvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:24:20.364472 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.364452    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" (UID: "a520fe86-fbd5-4b4f-aa79-e84dcafd4a68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:24:20.463083 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.463053    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47nvq\" (UniqueName: \"kubernetes.io/projected/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-kube-api-access-47nvq\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:24:20.463083 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.463079    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:24:20.463083 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.463088    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:24:20.714250 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.714162    2558 generic.go:356] "Generic (PLEG): container finished" podID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" containerID="db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1" exitCode=0
Jul 17 19:24:20.714250 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.714209    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"
Jul 17 19:24:20.714460 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.714244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" event={"ID":"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68","Type":"ContainerDied","Data":"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"}
Jul 17 19:24:20.714460 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.714306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522" event={"ID":"a520fe86-fbd5-4b4f-aa79-e84dcafd4a68","Type":"ContainerDied","Data":"e9d64c02b1eaaae28a2d2091f9036a2e2b54d821ee5a3b3f60f2be1eb52b3f62"}
Jul 17 19:24:20.714460 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.714356    2558 scope.go:117] "RemoveContainer" containerID="db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"
Jul 17 19:24:20.723741 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.723723    2558 scope.go:117] "RemoveContainer" containerID="db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"
Jul 17 19:24:20.724006 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:20.723990    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1\": container with ID starting with db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1 not found: ID does not exist" containerID="db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"
Jul 17 19:24:20.724051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.724012    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1"} err="failed to get container status \"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1\": rpc error: code = NotFound desc = could not find container \"db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1\": container with ID starting with db889630e3f1f2dc538feac9ad29a61e7caa1273200320610c974f3ad8bc7fc1 not found: ID does not exist"
Jul 17 19:24:20.748053 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.748029    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:24:20.760971 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.760948    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8fe1d7-55d586f956-lq522"]
Jul 17 19:24:20.772847 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:20.772820    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" path="/var/lib/kubelet/pods/a520fe86-fbd5-4b4f-aa79-e84dcafd4a68/volumes"
Jul 17 19:24:24.556857 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:24.556820    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"]
Jul 17 19:24:24.566619 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:24.566595    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fe1d7-q9vwt"]
Jul 17 19:24:24.772990 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:24.772960    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" path="/var/lib/kubelet/pods/f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16/volumes"
Jul 17 19:24:34.874877 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:34.874845    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:24:34.877024 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:34.876995    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:24:36.886420 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886387    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886740    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" containerName="revoke-keys"
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886755    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" containerName="revoke-keys"
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886769    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" containerName="maas-api"
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886775    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" containerName="maas-api"
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886843    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a520fe86-fbd5-4b4f-aa79-e84dcafd4a68" containerName="maas-api"
Jul 17 19:24:36.886900 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.886852    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bf8db8-37a3-48f0-ad2c-62fc7fd85b16" containerName="revoke-keys"
Jul 17 19:24:36.891388 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.891366    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:36.903272 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.903250    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-14077d-serving-cert\""
Jul 17 19:24:36.904550 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.904528    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cxfjz\""
Jul 17 19:24:36.904821 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.904803    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 19:24:36.904883 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.904803    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 19:24:36.924310 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:36.924283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:24:37.019957 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.019922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.020124 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.019994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.020124 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.020083    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww89\" (UniqueName: \"kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.120702 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.120663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.120883 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.120731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.120883 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.120771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww89\" (UniqueName: \"kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.120959 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:37.120890    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-serving-cert: secret "maas-api-e2e-shared-a-14077d-serving-cert" not found
Jul 17 19:24:37.120959 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:37.120957    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls podName:99371612-6969-4b18-84e2-463aa67a5b15 nodeName:}" failed. No retries permitted until 2026-07-17 19:24:37.620941162 +0000 UTC m=+2403.383371400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls") pod "maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" (UID: "99371612-6969-4b18-84e2-463aa67a5b15") : secret "maas-api-e2e-shared-a-14077d-serving-cert" not found
Jul 17 19:24:37.121346 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.121301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.140683 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.140603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww89\" (UniqueName: \"kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.625191 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.625151    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.627841 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.627807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") pod \"maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.804944 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.804897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:37.953002 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:37.952976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:24:37.955276 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:24:37.955249    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99371612_6969_4b18_84e2_463aa67a5b15.slice/crio-075ac995f0c68a709a8f3b0468d64d9b9412cc6ce20937431144d1f080a839b5 WatchSource:0}: Error finding container 075ac995f0c68a709a8f3b0468d64d9b9412cc6ce20937431144d1f080a839b5: Status 404 returned error can't find the container with id 075ac995f0c68a709a8f3b0468d64d9b9412cc6ce20937431144d1f080a839b5
Jul 17 19:24:38.805024 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:38.804990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" event={"ID":"99371612-6969-4b18-84e2-463aa67a5b15","Type":"ContainerStarted","Data":"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"}
Jul 17 19:24:38.805218 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:38.805052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" event={"ID":"99371612-6969-4b18-84e2-463aa67a5b15","Type":"ContainerStarted","Data":"075ac995f0c68a709a8f3b0468d64d9b9412cc6ce20937431144d1f080a839b5"}
Jul 17 19:24:38.805218 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:38.805081    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:38.824750 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:38.824709    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" podStartSLOduration=2.488331185 podStartE2EDuration="2.824694783s" totalImagesPullingTime="336.363598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:24:38.824073202 +0000 UTC m=+2404.586503473" watchObservedRunningTime="2026-07-17 19:24:38.824694783 +0000 UTC m=+2404.587125037"
Jul 17 19:24:44.816521 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:44.816490    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:24:44.967150 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:44.967117    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:24:44.970618 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:44.970598    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:44.973637 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:44.973618    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-2dc31c-serving-cert\""
Jul 17 19:24:44.986409 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:44.986377    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:24:45.094724 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.094638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pc7k\" (UniqueName: \"kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.094895 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.094740    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.094895 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.094765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.195234 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.195202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pc7k\" (UniqueName: \"kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.195462 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.195259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.195462 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.195283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.195462 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:45.195409    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-serving-cert: secret "maas-api-e2e-shared-b-2dc31c-serving-cert" not found
Jul 17 19:24:45.195652 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:45.195467    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls podName:4890e74d-c510-4c5f-9602-ca6ce5332d83 nodeName:}" failed. No retries permitted until 2026-07-17 19:24:45.695449599 +0000 UTC m=+2411.457879836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls") pod "maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" (UID: "4890e74d-c510-4c5f-9602-ca6ce5332d83") : secret "maas-api-e2e-shared-b-2dc31c-serving-cert" not found
Jul 17 19:24:45.196002 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.195980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.216217 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.216186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pc7k\" (UniqueName: \"kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.700311 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:45.700266    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:45.700545 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:45.700419    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-serving-cert: secret "maas-api-e2e-shared-b-2dc31c-serving-cert" not found
Jul 17 19:24:45.700545 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:24:45.700514    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls podName:4890e74d-c510-4c5f-9602-ca6ce5332d83 nodeName:}" failed. No retries permitted until 2026-07-17 19:24:46.700493527 +0000 UTC m=+2412.462923764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls") pod "maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" (UID: "4890e74d-c510-4c5f-9602-ca6ce5332d83") : secret "maas-api-e2e-shared-b-2dc31c-serving-cert" not found
Jul 17 19:24:46.709099 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:46.709059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:46.711759 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:46.711724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:46.780660 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:46.780629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:46.935380 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:46.935347    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:24:46.938803 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:24:46.938776    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4890e74d_c510_4c5f_9602_ca6ce5332d83.slice/crio-2c854156378dd6ff14df48537564735a325041b53bedd25690c3b6b5ee17a15a WatchSource:0}: Error finding container 2c854156378dd6ff14df48537564735a325041b53bedd25690c3b6b5ee17a15a: Status 404 returned error can't find the container with id 2c854156378dd6ff14df48537564735a325041b53bedd25690c3b6b5ee17a15a
Jul 17 19:24:47.851207 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:47.851167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" event={"ID":"4890e74d-c510-4c5f-9602-ca6ce5332d83","Type":"ContainerStarted","Data":"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"}
Jul 17 19:24:47.851641 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:47.851224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" event={"ID":"4890e74d-c510-4c5f-9602-ca6ce5332d83","Type":"ContainerStarted","Data":"2c854156378dd6ff14df48537564735a325041b53bedd25690c3b6b5ee17a15a"}
Jul 17 19:24:47.851641 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:47.851278    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:24:47.886691 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:47.886643    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" podStartSLOduration=3.545619213 podStartE2EDuration="3.886628685s" totalImagesPullingTime="341.009472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:24:47.886364162 +0000 UTC m=+2413.648794442" watchObservedRunningTime="2026-07-17 19:24:47.886628685 +0000 UTC m=+2413.649058993"
Jul 17 19:24:53.862222 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:24:53.862191    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:25:35.046834 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:25:35.046802    2558 scope.go:117] "RemoveContainer" containerID="1e12bfcee417f3eaf89e8a3387f184c70267fbba9b084bf9cc8bbfdb9022cb63"
Jul 17 19:27:35.073331 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:27:35.073283    2558 scope.go:117] "RemoveContainer" containerID="e4a398839bcd29f432ba170a138c9df43797487effe45fb60bd34ecbba30d034"
Jul 17 19:28:10.728727 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.728690    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:28:10.736144 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.734149    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.740965 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.740942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14077d\"/\"kube-root-ca.crt\""
Jul 17 19:28:10.741998 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.741977    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14077d\"/\"sub-test-model-14077d-kserve-self-signed-certs\""
Jul 17 19:28:10.742108 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.741984    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14077d\"/\"openshift-service-ca.crt\""
Jul 17 19:28:10.742108 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.742029    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14077d\"/\"default-dockercfg-fzqhn\""
Jul 17 19:28:10.754746 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.754714    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:28:10.890868 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.890834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.891051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.890876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.891051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.890956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.891051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.891015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.891162 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.891075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.891162 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.891104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbpt\" (UniqueName: \"kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.991873 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.991782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbpt\" (UniqueName: \"kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.991873 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.991830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992062 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.991959    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992062 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992161 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992161 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992246 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992173    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992437 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.992510 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.992418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.994539 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.994521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:10.994863 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:10.994829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:11.006196 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.006172    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbpt\" (UniqueName: \"kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt\") pod \"sub-test-model-14077d-kserve-677cbf5f5c-fp2qt\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") " pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:11.051103 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.051066    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:11.242962 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.242883    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:28:11.246715 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:28:11.246688    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f30201f_8a0d_4c05_9b7c_f7e834c17e5a.slice/crio-b7d22824e02bcd085157ad36845984d3cb20e83fac9f12454890e5818773b037 WatchSource:0}: Error finding container b7d22824e02bcd085157ad36845984d3cb20e83fac9f12454890e5818773b037: Status 404 returned error can't find the container with id b7d22824e02bcd085157ad36845984d3cb20e83fac9f12454890e5818773b037
Jul 17 19:28:11.859666 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.859626    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" event={"ID":"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a","Type":"ContainerStarted","Data":"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"}
Jul 17 19:28:11.860064 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.859675    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" event={"ID":"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a","Type":"ContainerStarted","Data":"b7d22824e02bcd085157ad36845984d3cb20e83fac9f12454890e5818773b037"}
Jul 17 19:28:11.894444 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:11.894391    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" podStartSLOduration=1.894374083 podStartE2EDuration="1.894374083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:28:11.890097176 +0000 UTC m=+2617.652527435" watchObservedRunningTime="2026-07-17 19:28:11.894374083 +0000 UTC m=+2617.656804342"
Jul 17 19:28:21.051434 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:21.051396    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:21.051434 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:21.051453    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:21.064201 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:21.064171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:21.921595 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:21.921551    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:28:35.094336 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:28:35.094300    2558 scope.go:117] "RemoveContainer" containerID="69e33243db312a3fab53908d24b1b508b2ab3c81b06d526a430338a5eac9bbcc"
Jul 17 19:29:25.451282 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:25.451247    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:29:34.904979 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:34.904893    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:29:34.907141 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:34.907117    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:29:35.114271 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:35.114236    2558 scope.go:117] "RemoveContainer" containerID="c060a9900b907bfed65427e490c17b8eefd65bee12ace42eabf11cb924fce715"
Jul 17 19:29:40.472875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.472829    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" podUID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" containerName="main" containerID="cri-o://07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a" gracePeriod=45
Jul 17 19:29:40.723132 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.723074    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:29:40.845879 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.845849    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846061 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.845896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846061 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.845922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846061 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.845940    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkbpt\" (UniqueName: \"kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846061 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.845967    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846275 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm\") pod \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\" (UID: \"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a\") "
Jul 17 19:29:40.846275 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846180    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:29:40.846275 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home" (OuterVolumeSpecName: "home") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:29:40.846275 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846263    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:29:40.846511 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846498    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tmp-dir\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:40.846568 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846513    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-home\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:40.846568 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.846525    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-model-cache\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:40.848403 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.848381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:29:40.848502 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.848408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm" (OuterVolumeSpecName: "dshm") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 19:29:40.848502 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.848421    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt" (OuterVolumeSpecName: "kube-api-access-bkbpt") pod "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" (UID: "2f30201f-8a0d-4c05-9b7c-f7e834c17e5a"). InnerVolumeSpecName "kube-api-access-bkbpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:29:40.947299 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.947267    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-dshm\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:40.947299 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.947296    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-tls-certs\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:40.947299 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:40.947307    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkbpt\" (UniqueName: \"kubernetes.io/projected/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a-kube-api-access-bkbpt\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:29:41.299342 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.299278    2558 generic.go:356] "Generic (PLEG): container finished" podID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" containerID="07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a" exitCode=0
Jul 17 19:29:41.299342 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.299338    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"
Jul 17 19:29:41.299579 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.299362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" event={"ID":"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a","Type":"ContainerDied","Data":"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"}
Jul 17 19:29:41.299579 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.299406    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt" event={"ID":"2f30201f-8a0d-4c05-9b7c-f7e834c17e5a","Type":"ContainerDied","Data":"b7d22824e02bcd085157ad36845984d3cb20e83fac9f12454890e5818773b037"}
Jul 17 19:29:41.299579 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.299424    2558 scope.go:117] "RemoveContainer" containerID="07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"
Jul 17 19:29:41.308906 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.308889    2558 scope.go:117] "RemoveContainer" containerID="07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"
Jul 17 19:29:41.309196 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:29:41.309178    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a\": container with ID starting with 07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a not found: ID does not exist" containerID="07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"
Jul 17 19:29:41.309249 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.309205    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a"} err="failed to get container status \"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a\": rpc error: code = NotFound desc = could not find container \"07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a\": container with ID starting with 07e0811f36233a5b1fbedff72dea4cbfdbab732d0c41a6e99c254fb4ef71677a not found: ID does not exist"
Jul 17 19:29:41.327786 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.327759    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:29:41.329875 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:41.329854    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-14077d/sub-test-model-14077d-kserve-677cbf5f5c-fp2qt"]
Jul 17 19:29:42.773169 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:29:42.773135    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" path="/var/lib/kubelet/pods/2f30201f-8a0d-4c05-9b7c-f7e834c17e5a/volumes"
Jul 17 19:30:00.199203 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.199161    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"]
Jul 17 19:30:00.199844 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.199729    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" containerName="main"
Jul 17 19:30:00.199844 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.199747    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" containerName="main"
Jul 17 19:30:00.199844 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.199825    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f30201f-8a0d-4c05-9b7c-f7e834c17e5a" containerName="main"
Jul 17 19:30:00.203411 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.203389    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"]
Jul 17 19:30:00.203544 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.203523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"
Jul 17 19:30:00.206758 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.206741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"
Jul 17 19:30:00.209869 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.209850    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:30:00.213854 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.213838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"
Jul 17 19:30:00.220599 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.220571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"
Jul 17 19:30:00.225476 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.225455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"]
Jul 17 19:30:00.271366 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.271297    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"]
Jul 17 19:30:00.380569 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.380534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"]
Jul 17 19:30:00.384367 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:30:00.384335    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2812ab7_76eb_40f5_9f64_e7ff035c66d4.slice/crio-72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93 WatchSource:0}: Error finding container 72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93: Status 404 returned error can't find the container with id 72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93
Jul 17 19:30:00.392138 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.392116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt" event={"ID":"d2812ab7-76eb-40f5-9f64-e7ff035c66d4","Type":"ContainerStarted","Data":"72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93"}
Jul 17 19:30:00.415480 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:00.415452    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"]
Jul 17 19:30:00.423379 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:30:00.423350    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c4652c_268e_487f_b7e7_1c2039fd51bb.slice/crio-2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09 WatchSource:0}: Error finding container 2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09: Status 404 returned error can't find the container with id 2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09
Jul 17 19:30:01.399063 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:01.399026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5" event={"ID":"e2c4652c-268e-487f-b7e7-1c2039fd51bb","Type":"ContainerStarted","Data":"7ea0317ac5f24eb838caad127976c04ae825347167c184fbfff5a4b014c676de"}
Jul 17 19:30:01.399466 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:01.399096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5" event={"ID":"e2c4652c-268e-487f-b7e7-1c2039fd51bb","Type":"ContainerStarted","Data":"2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09"}
Jul 17 19:30:01.400939 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:01.400905    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt" event={"ID":"d2812ab7-76eb-40f5-9f64-e7ff035c66d4","Type":"ContainerStarted","Data":"8e4e0f4fc4c892442b0df095f8e73cfc417c229cc1d7c0b14f7ec2694b435422"}
Jul 17 19:30:01.420756 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:01.420714    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5" podStartSLOduration=1.420696338 podStartE2EDuration="1.420696338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:01.416370262 +0000 UTC m=+2727.178800543" watchObservedRunningTime="2026-07-17 19:30:01.420696338 +0000 UTC m=+2727.183126598"
Jul 17 19:30:01.435002 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:01.434955    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt" podStartSLOduration=1.434943029 podStartE2EDuration="1.434943029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:30:01.433183097 +0000 UTC m=+2727.195613357" watchObservedRunningTime="2026-07-17 19:30:01.434943029 +0000 UTC m=+2727.197373266"
Jul 17 19:30:02.408058 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:02.408019    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2812ab7-76eb-40f5-9f64-e7ff035c66d4" containerID="8e4e0f4fc4c892442b0df095f8e73cfc417c229cc1d7c0b14f7ec2694b435422" exitCode=0
Jul 17 19:30:02.408570 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:02.408106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt" event={"ID":"d2812ab7-76eb-40f5-9f64-e7ff035c66d4","Type":"ContainerDied","Data":"8e4e0f4fc4c892442b0df095f8e73cfc417c229cc1d7c0b14f7ec2694b435422"}
Jul 17 19:30:02.409858 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:02.409829    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2c4652c-268e-487f-b7e7-1c2039fd51bb" containerID="7ea0317ac5f24eb838caad127976c04ae825347167c184fbfff5a4b014c676de" exitCode=0
Jul 17 19:30:02.409997 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:02.409906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5" event={"ID":"e2c4652c-268e-487f-b7e7-1c2039fd51bb","Type":"ContainerDied","Data":"7ea0317ac5f24eb838caad127976c04ae825347167c184fbfff5a4b014c676de"}
Jul 17 19:30:03.549209 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:03.549183    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"
Jul 17 19:30:03.587966 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:03.587935    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"
Jul 17 19:30:04.421468 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.421435    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt"
Jul 17 19:30:04.421663 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.421474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738610-lvtlt" event={"ID":"d2812ab7-76eb-40f5-9f64-e7ff035c66d4","Type":"ContainerDied","Data":"72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93"}
Jul 17 19:30:04.421663 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.421527    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72491a0bdb9df9fa60093cc4c27973f477108b18516490bac00835c368866f93"
Jul 17 19:30:04.422923 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.422905    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"
Jul 17 19:30:04.422923 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.422909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5" event={"ID":"e2c4652c-268e-487f-b7e7-1c2039fd51bb","Type":"ContainerDied","Data":"2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09"}
Jul 17 19:30:04.423126 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:04.422948    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a66ed1920964315a7b8cdbdc84283c862536eb7e03fc6961409087c9f878f09"
Jul 17 19:30:35.133801 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:35.133766    2558 scope.go:117] "RemoveContainer" containerID="29f6780fa5462dbbf8a6ced617945b78a6a16ca5cb3adc8d8fa669c9be916f70"
Jul 17 19:30:35.142367 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:30:35.142336    2558 scope.go:117] "RemoveContainer" containerID="1153724640da6ef932457531745f5fea49436ea919acde5eb4604999c791a2f9"
Jul 17 19:31:09.218181 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.218145    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:31:09.219111 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.219055    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" podUID="99371612-6969-4b18-84e2-463aa67a5b15" containerName="maas-api" containerID="cri-o://2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836" gracePeriod=30
Jul 17 19:31:09.484407 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.484337    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:31:09.571457 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.571427    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") pod \"99371612-6969-4b18-84e2-463aa67a5b15\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") "
Jul 17 19:31:09.571633 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.571501    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca\") pod \"99371612-6969-4b18-84e2-463aa67a5b15\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") "
Jul 17 19:31:09.571633 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.571538    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ww89\" (UniqueName: \"kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89\") pod \"99371612-6969-4b18-84e2-463aa67a5b15\" (UID: \"99371612-6969-4b18-84e2-463aa67a5b15\") "
Jul 17 19:31:09.571879 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.571857    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99371612-6969-4b18-84e2-463aa67a5b15" (UID: "99371612-6969-4b18-84e2-463aa67a5b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:09.573812 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.573788    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89" (OuterVolumeSpecName: "kube-api-access-6ww89") pod "99371612-6969-4b18-84e2-463aa67a5b15" (UID: "99371612-6969-4b18-84e2-463aa67a5b15"). InnerVolumeSpecName "kube-api-access-6ww89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:31:09.573886 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.573823    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99371612-6969-4b18-84e2-463aa67a5b15" (UID: "99371612-6969-4b18-84e2-463aa67a5b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:31:09.673003 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.672965    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99371612-6969-4b18-84e2-463aa67a5b15-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:09.673003 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.672997    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww89\" (UniqueName: \"kubernetes.io/projected/99371612-6969-4b18-84e2-463aa67a5b15-kube-api-access-6ww89\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:09.673003 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.673009    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99371612-6969-4b18-84e2-463aa67a5b15-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:09.733223 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.733183    2558 generic.go:356] "Generic (PLEG): container finished" podID="99371612-6969-4b18-84e2-463aa67a5b15" containerID="2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836" exitCode=0
Jul 17 19:31:09.733439 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.733234    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"
Jul 17 19:31:09.733439 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.733262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" event={"ID":"99371612-6969-4b18-84e2-463aa67a5b15","Type":"ContainerDied","Data":"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"}
Jul 17 19:31:09.733439 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.733356    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9" event={"ID":"99371612-6969-4b18-84e2-463aa67a5b15","Type":"ContainerDied","Data":"075ac995f0c68a709a8f3b0468d64d9b9412cc6ce20937431144d1f080a839b5"}
Jul 17 19:31:09.733439 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.733391    2558 scope.go:117] "RemoveContainer" containerID="2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"
Jul 17 19:31:09.743165 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.743144    2558 scope.go:117] "RemoveContainer" containerID="2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"
Jul 17 19:31:09.743495 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:31:09.743473    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836\": container with ID starting with 2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836 not found: ID does not exist" containerID="2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"
Jul 17 19:31:09.743549 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.743504    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836"} err="failed to get container status \"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836\": rpc error: code = NotFound desc = could not find container \"2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836\": container with ID starting with 2e9cc0e96acd8072d8457dc767959f77e0303647460334acc1d806895766e836 not found: ID does not exist"
Jul 17 19:31:09.755234 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.755202    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:31:09.758518 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:09.758496    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-14077d-c64c9574d-wxzm9"]
Jul 17 19:31:10.772902 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:10.772872    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99371612-6969-4b18-84e2-463aa67a5b15" path="/var/lib/kubelet/pods/99371612-6969-4b18-84e2-463aa67a5b15/volumes"
Jul 17 19:31:21.845762 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.845727    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"]
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846058    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99371612-6969-4b18-84e2-463aa67a5b15" containerName="maas-api"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846069    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99371612-6969-4b18-84e2-463aa67a5b15" containerName="maas-api"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846081    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2812ab7-76eb-40f5-9f64-e7ff035c66d4" containerName="cleanup"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846087    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2812ab7-76eb-40f5-9f64-e7ff035c66d4" containerName="cleanup"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846093    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c4652c-268e-487f-b7e7-1c2039fd51bb" containerName="cleanup"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846098    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c4652c-268e-487f-b7e7-1c2039fd51bb" containerName="cleanup"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846155    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2812ab7-76eb-40f5-9f64-e7ff035c66d4" containerName="cleanup"
Jul 17 19:31:21.846160 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846164    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c4652c-268e-487f-b7e7-1c2039fd51bb" containerName="cleanup"
Jul 17 19:31:21.846446 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.846174    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99371612-6969-4b18-84e2-463aa67a5b15" containerName="maas-api"
Jul 17 19:31:21.849116 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.849099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:21.851683 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.851659    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwjqx\""
Jul 17 19:31:21.858154 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.858126    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"]
Jul 17 19:31:21.991193 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:21.991149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx\" (UID: \"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:22.091993 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.091956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx\" (UID: \"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:22.092639 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.092622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx\" (UID: \"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:22.161810 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.161723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:22.317726 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.317696    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"]
Jul 17 19:31:22.317860 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:31:22.317813    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f37fb9_6ed2_4dff_ba9a_17fabe708c38.slice/crio-9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e WatchSource:0}: Error finding container 9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e: Status 404 returned error can't find the container with id 9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e
Jul 17 19:31:22.794238 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.794143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx" event={"ID":"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38","Type":"ContainerStarted","Data":"c7b35585ec776a242250bf3707f44dc068b9b796e313fe5e3d94ba0ca5002547"}
Jul 17 19:31:22.794238 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.794207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx" event={"ID":"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38","Type":"ContainerStarted","Data":"9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e"}
Jul 17 19:31:22.813595 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:22.813545    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx" podStartSLOduration=1.813528402 podStartE2EDuration="1.813528402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:31:22.812427207 +0000 UTC m=+2808.574857511" watchObservedRunningTime="2026-07-17 19:31:22.813528402 +0000 UTC m=+2808.575958661"
Jul 17 19:31:23.801281 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:23.801243    2558 generic.go:356] "Generic (PLEG): container finished" podID="e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" containerID="c7b35585ec776a242250bf3707f44dc068b9b796e313fe5e3d94ba0ca5002547" exitCode=0
Jul 17 19:31:23.801829 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:23.801306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx" event={"ID":"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38","Type":"ContainerDied","Data":"c7b35585ec776a242250bf3707f44dc068b9b796e313fe5e3d94ba0ca5002547"}
Jul 17 19:31:24.943195 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:24.943170    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:25.119386 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.119277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca\") pod \"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38\" (UID: \"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38\") "
Jul 17 19:31:25.119641 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.119619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" (UID: "e9f37fb9-6ed2-4dff-ba9a-17fabe708c38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:25.220304 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.220266    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38-maas-api-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:25.812020 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.811985    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx" event={"ID":"e9f37fb9-6ed2-4dff-ba9a-17fabe708c38","Type":"ContainerDied","Data":"9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e"}
Jul 17 19:31:25.812020 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.812004    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"
Jul 17 19:31:25.812247 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:25.812034    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f349a8eaf97db0ee5720913e03ec1805ac5656971cb7858c3ce8cfbae54e24e"
Jul 17 19:31:32.431124 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.431087    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:31:32.431634 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.431447    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" podUID="4890e74d-c510-4c5f-9602-ca6ce5332d83" containerName="maas-api" containerID="cri-o://e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e" gracePeriod=30
Jul 17 19:31:32.494643 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.494613    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"]
Jul 17 19:31:32.498361 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.498333    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-2dc31c-29738610-v9xz5"]
Jul 17 19:31:32.682036 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.681967    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:31:32.690537 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.690512    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") pod \"4890e74d-c510-4c5f-9602-ca6ce5332d83\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") "
Jul 17 19:31:32.690686 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.690593    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca\") pod \"4890e74d-c510-4c5f-9602-ca6ce5332d83\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") "
Jul 17 19:31:32.690686 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.690628    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pc7k\" (UniqueName: \"kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k\") pod \"4890e74d-c510-4c5f-9602-ca6ce5332d83\" (UID: \"4890e74d-c510-4c5f-9602-ca6ce5332d83\") "
Jul 17 19:31:32.690946 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.690918    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4890e74d-c510-4c5f-9602-ca6ce5332d83" (UID: "4890e74d-c510-4c5f-9602-ca6ce5332d83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 19:31:32.692879 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.692854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4890e74d-c510-4c5f-9602-ca6ce5332d83" (UID: "4890e74d-c510-4c5f-9602-ca6ce5332d83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 19:31:32.692965 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.692875    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k" (OuterVolumeSpecName: "kube-api-access-6pc7k") pod "4890e74d-c510-4c5f-9602-ca6ce5332d83" (UID: "4890e74d-c510-4c5f-9602-ca6ce5332d83"). InnerVolumeSpecName "kube-api-access-6pc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 19:31:32.773061 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.773031    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2c4652c-268e-487f-b7e7-1c2039fd51bb" path="/var/lib/kubelet/pods/e2c4652c-268e-487f-b7e7-1c2039fd51bb/volumes"
Jul 17 19:31:32.791677 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.791642    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4890e74d-c510-4c5f-9602-ca6ce5332d83-openshift-service-ca\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:32.791677 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.791674    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pc7k\" (UniqueName: \"kubernetes.io/projected/4890e74d-c510-4c5f-9602-ca6ce5332d83-kube-api-access-6pc7k\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:32.791677 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.791687    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4890e74d-c510-4c5f-9602-ca6ce5332d83-maas-api-tls\") on node \"ip-10-0-138-140.ec2.internal\" DevicePath \"\""
Jul 17 19:31:32.847221 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.847189    2558 generic.go:356] "Generic (PLEG): container finished" podID="4890e74d-c510-4c5f-9602-ca6ce5332d83" containerID="e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e" exitCode=0
Jul 17 19:31:32.847431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.847240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"
Jul 17 19:31:32.847431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.847290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" event={"ID":"4890e74d-c510-4c5f-9602-ca6ce5332d83","Type":"ContainerDied","Data":"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"}
Jul 17 19:31:32.847431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.847366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6" event={"ID":"4890e74d-c510-4c5f-9602-ca6ce5332d83","Type":"ContainerDied","Data":"2c854156378dd6ff14df48537564735a325041b53bedd25690c3b6b5ee17a15a"}
Jul 17 19:31:32.847431 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.847396    2558 scope.go:117] "RemoveContainer" containerID="e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"
Jul 17 19:31:32.856167 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.856145    2558 scope.go:117] "RemoveContainer" containerID="e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"
Jul 17 19:31:32.856470 ip-10-0-138-140 kubenswrapper[2558]: E0717 19:31:32.856451    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e\": container with ID starting with e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e not found: ID does not exist" containerID="e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"
Jul 17 19:31:32.856538 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.856478    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e"} err="failed to get container status \"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e\": rpc error: code = NotFound desc = could not find container \"e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e\": container with ID starting with e6c22d593a9d60e146cf09654127160b3151339750375d7b4733d24b0e023e8e not found: ID does not exist"
Jul 17 19:31:32.872707 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.872681    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:31:32.876668 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:32.876647    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-2dc31c-7dc5c678c6-k49h6"]
Jul 17 19:31:34.773373 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:34.773314    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4890e74d-c510-4c5f-9602-ca6ce5332d83" path="/var/lib/kubelet/pods/4890e74d-c510-4c5f-9602-ca6ce5332d83/volumes"
Jul 17 19:31:36.945281 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:36.945241    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"]
Jul 17 19:31:36.948581 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:36.948554    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-2dc31c-z44sx"]
Jul 17 19:31:38.772978 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:38.772949    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" path="/var/lib/kubelet/pods/e9f37fb9-6ed2-4dff-ba9a-17fabe708c38/volumes"
Jul 17 19:31:51.880857 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:51.880826    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68969b7c46-jzkbx_015b05c6-f57a-48e5-be54-d87ecde170f4/copy-manifests/0.log"
Jul 17 19:31:51.902457 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:51.902416    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68969b7c46-jzkbx_015b05c6-f57a-48e5-be54-d87ecde170f4/manager/0.log"
Jul 17 19:31:52.437519 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:52.437490    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738580-bpzlc_0b222375-a939-4653-b8f9-992a895bb941/cleanup/0.log"
Jul 17 19:31:52.699463 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:52.699378    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738610-lvtlt_d2812ab7-76eb-40f5-9f64-e7ff035c66d4/cleanup/0.log"
Jul 17 19:31:54.904947 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:54.904921    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cgh2f_c1ed5bae-be7f-43a1-9253-2ad684ddc5d4/manager/0.log"
Jul 17 19:31:55.668504 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:55.668468    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5jsdm_4fad983e-b65c-4529-94d1-23890da216fb/discovery/0.log"
Jul 17 19:31:56.396103 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:56.396063    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c6cb4dc5b-f5hkm_c4898006-61dc-42b4-a9d6-b3022d1adc4f/router/0.log"
Jul 17 19:31:56.769716 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:56.769624    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-vnkw9_be63392d-9c1b-43f2-8af2-96f4d39b5c67/main/0.log"
Jul 17 19:31:56.889802 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:31:56.889764    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-56zdh_3ec2fe88-76e6-4d17-9a6f-4aee17650144/main/0.log"
Jul 17 19:32:06.565354 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:06.565307    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wv5n9_3761999b-61c5-40c7-8d95-5c228ce395b0/global-pull-secret-syncer/0.log"
Jul 17 19:32:06.737746 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:06.737708    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vtlz6_2dc37e0f-36bd-4e70-b105-fc17931f9699/konnectivity-agent/0.log"
Jul 17 19:32:06.845148 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:06.845063    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-140.ec2.internal_623d991e054ce9bc2a7b170b88241ceb/haproxy/0.log"
Jul 17 19:32:13.310012 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:13.309983    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cgh2f_c1ed5bae-be7f-43a1-9253-2ad684ddc5d4/manager/0.log"
Jul 17 19:32:15.349594 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.349558    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-b4pxl_42b75940-c085-4f8c-bce5-4d23c1f6a7b2/cluster-monitoring-operator/0.log"
Jul 17 19:32:15.716872 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.716790    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txcns_ba5cf590-e4bf-45a3-a9a0-45741a7301c8/node-exporter/0.log"
Jul 17 19:32:15.745736 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.745703    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txcns_ba5cf590-e4bf-45a3-a9a0-45741a7301c8/kube-rbac-proxy/0.log"
Jul 17 19:32:15.792579 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.792555    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-txcns_ba5cf590-e4bf-45a3-a9a0-45741a7301c8/init-textfile/0.log"
Jul 17 19:32:15.802936 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.802900    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"]
Jul 17 19:32:15.803265 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803253    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4890e74d-c510-4c5f-9602-ca6ce5332d83" containerName="maas-api"
Jul 17 19:32:15.803308 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803267    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4890e74d-c510-4c5f-9602-ca6ce5332d83" containerName="maas-api"
Jul 17 19:32:15.803308 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803282    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" containerName="revoke-keys"
Jul 17 19:32:15.803308 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803288    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" containerName="revoke-keys"
Jul 17 19:32:15.803442 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803352    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4890e74d-c510-4c5f-9602-ca6ce5332d83" containerName="maas-api"
Jul 17 19:32:15.803442 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.803362    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f37fb9-6ed2-4dff-ba9a-17fabe708c38" containerName="revoke-keys"
Jul 17 19:32:15.806281 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.806262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.808821 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.808791    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sqxlf\"/\"kube-root-ca.crt\""
Jul 17 19:32:15.809832 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.809809    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sqxlf\"/\"openshift-service-ca.crt\""
Jul 17 19:32:15.809832 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.809821    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sqxlf\"/\"default-dockercfg-r7slb\""
Jul 17 19:32:15.820017 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.819990    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"]
Jul 17 19:32:15.881295 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.881253    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-proc\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.881295 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.881303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-sys\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.881572 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.881387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rnm\" (UniqueName: \"kubernetes.io/projected/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-kube-api-access-r9rnm\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.881572 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.881435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-podres\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.881572 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.881466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-lib-modules\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.931618 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.931584    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/prometheus/0.log"
Jul 17 19:32:15.955786 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.955737    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/config-reloader/0.log"
Jul 17 19:32:15.980543 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.980457    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/thanos-sidecar/0.log"
Jul 17 19:32:15.982152 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-proc\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982247 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-sys\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982247 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rnm\" (UniqueName: \"kubernetes.io/projected/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-kube-api-access-r9rnm\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982247 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-podres\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982247 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-lib-modules\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982432 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-sys\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982432 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982242    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-proc\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982432 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-lib-modules\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.982432 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.982390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-podres\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:15.991234 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:15.991210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rnm\" (UniqueName: \"kubernetes.io/projected/cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc-kube-api-access-r9rnm\") pod \"perf-node-gather-daemonset-kgfk4\" (UID: \"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc\") " pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:16.005635 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.005612    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/kube-rbac-proxy-web/0.log"
Jul 17 19:32:16.032500 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.032475    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/kube-rbac-proxy/0.log"
Jul 17 19:32:16.059003 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.058976    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/kube-rbac-proxy-thanos/0.log"
Jul 17 19:32:16.084573 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.084547    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1790c087-12de-43a8-9bec-c6ecca1e8054/init-config-reloader/0.log"
Jul 17 19:32:16.125442 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.125404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:16.258932 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:16.258900    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"]
Jul 17 19:32:16.261877 ip-10-0-138-140 kubenswrapper[2558]: W0717 19:32:16.261851    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb3058c8_6efb_487b_8dcc_6e8d9f0aa4cc.slice/crio-e7d836ea7c3c7ea5eaff2bd7964d4631cb2cb6c309e9108a6c5dbbb24451e9c5 WatchSource:0}: Error finding container e7d836ea7c3c7ea5eaff2bd7964d4631cb2cb6c309e9108a6c5dbbb24451e9c5: Status 404 returned error can't find the container with id e7d836ea7c3c7ea5eaff2bd7964d4631cb2cb6c309e9108a6c5dbbb24451e9c5
Jul 17 19:32:17.056491 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:17.056454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4" event={"ID":"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc","Type":"ContainerStarted","Data":"86875ae602ca3458956920654cd6ff6a46fc597483451c71e600cfc6142d569e"}
Jul 17 19:32:17.056927 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:17.056502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4" event={"ID":"cb3058c8-6efb-487b-8dcc-6e8d9f0aa4cc","Type":"ContainerStarted","Data":"e7d836ea7c3c7ea5eaff2bd7964d4631cb2cb6c309e9108a6c5dbbb24451e9c5"}
Jul 17 19:32:17.056927 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:17.056572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:17.075610 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:17.075561    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4" podStartSLOduration=2.075543825 podStartE2EDuration="2.075543825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:32:17.075256121 +0000 UTC m=+2862.837686397" watchObservedRunningTime="2026-07-17 19:32:17.075543825 +0000 UTC m=+2862.837974086"
Jul 17 19:32:17.551238 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:17.551210    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-pp97k_d22bd50b-1802-473b-8dd7-1e5a96e7ff6b/networking-console-plugin/0.log"
Jul 17 19:32:18.168873 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:18.168845    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/1.log"
Jul 17 19:32:18.177119 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:18.177092    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-wrcwt_b4d66827-0a83-434c-b174-8224d6fa5830/console-operator/2.log"
Jul 17 19:32:19.466170 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:19.466141    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-ntj9r_cc2303f5-dba3-4879-969c-d31129b82416/volume-data-source-validator/0.log"
Jul 17 19:32:20.679181 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:20.679153    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hph6v_f8be0c3f-c344-4968-92da-6e028a539e6f/dns/0.log"
Jul 17 19:32:20.719701 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:20.719656    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hph6v_f8be0c3f-c344-4968-92da-6e028a539e6f/kube-rbac-proxy/0.log"
Jul 17 19:32:20.940059 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:20.939976    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcbkz_24ba022f-d700-4dea-83fe-3f00bd2bd1db/dns-node-resolver/0.log"
Jul 17 19:32:21.643153 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:21.643122    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84fb5db76b-f97n9_0a883a90-b8e3-46eb-a99d-eeded91debbd/registry/0.log"
Jul 17 19:32:21.691051 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:21.691019    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-htt78_f4c47689-ae80-4e61-bb80-f4cb8f3dfadf/node-ca/0.log"
Jul 17 19:32:23.074014 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:23.073986    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sqxlf/perf-node-gather-daemonset-kgfk4"
Jul 17 19:32:23.101976 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:23.101942    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5jsdm_4fad983e-b65c-4529-94d1-23890da216fb/discovery/0.log"
Jul 17 19:32:23.364596 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:23.364517    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c6cb4dc5b-f5hkm_c4898006-61dc-42b4-a9d6-b3022d1adc4f/router/0.log"
Jul 17 19:32:24.434739 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:24.434715    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xpshk_a7589901-91ef-4eeb-8d9b-c5c60afb2368/serve-healthcheck-canary/0.log"
Jul 17 19:32:25.106627 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:25.106592    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cm2gp_954c5632-fd22-4e9b-9536-b8f6dd21b5c0/insights-operator/0.log"
Jul 17 19:32:25.108147 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:25.108127    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-cm2gp_954c5632-fd22-4e9b-9536-b8f6dd21b5c0/insights-operator/1.log"
Jul 17 19:32:25.137499 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:25.137478    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qkzh_87336d0b-4178-4b44-b999-a44ed8434de8/kube-rbac-proxy/0.log"
Jul 17 19:32:25.161841 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:25.161817    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qkzh_87336d0b-4178-4b44-b999-a44ed8434de8/exporter/0.log"
Jul 17 19:32:25.187570 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:25.187545    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qkzh_87336d0b-4178-4b44-b999-a44ed8434de8/extractor/0.log"
Jul 17 19:32:27.350488 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:27.350450    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5jsdm_4fad983e-b65c-4529-94d1-23890da216fb/discovery/0.log"
Jul 17 19:32:27.506896 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:27.506787    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c6cb4dc5b-f5hkm_c4898006-61dc-42b4-a9d6-b3022d1adc4f/router/0.log"
Jul 17 19:32:28.671842 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:28.671810    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68969b7c46-jzkbx_015b05c6-f57a-48e5-be54-d87ecde170f4/manager/0.log"
Jul 17 19:32:28.709631 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:28.709606    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68969b7c46-jzkbx_015b05c6-f57a-48e5-be54-d87ecde170f4/copy-manifests/0.log"
Jul 17 19:32:28.910490 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:28.910460    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5jsdm_4fad983e-b65c-4529-94d1-23890da216fb/discovery/0.log"
Jul 17 19:32:29.120445 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:29.120419    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5c6cb4dc5b-f5hkm_c4898006-61dc-42b4-a9d6-b3022d1adc4f/router/0.log"
Jul 17 19:32:30.042410 ip-10-0-138-140 kubenswrapper[2558]: I0717 19:32:30.042374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76fdf749bb-vf2bt_46b4f2e4-ced5-48e7-a52b-a622514933ce/manager/0.log"