Jul 23 09:16:55.211196 ip-10-0-129-47 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.572004    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580042    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580059    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580063    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580066    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580069    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580071    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580074    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.647179 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580077    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580080    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580083    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580085    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580089    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580091    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580094    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580096    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580100    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580102    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580105    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580108    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580110    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580113    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580115    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580119    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580123    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580125    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580128    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.649417 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580131    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580134    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580136    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580139    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580143    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580147    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580151    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580154    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580157    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580160    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580164    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580168    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580171    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580174    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580177    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580180    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580183    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580186    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580189    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.651015 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580191    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580194    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580197    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580200    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580202    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580205    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580208    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580211    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580213    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580216    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580219    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580221    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580224    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580226    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580229    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580232    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580234    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580237    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580239    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580242    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.652454 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580246    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580249    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580252    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580254    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580257    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580259    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580262    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580264    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580267    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580269    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580272    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580274    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580277    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580280    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580284    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580286    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580289    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580291    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580294    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580297    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.654690 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580300    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580302    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580305    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580307    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580310    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580312    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580315    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580317    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580320    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580323    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580325    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580328    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580331    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.580334    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581025    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581032    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581035    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581038    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581042    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.655522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581045    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581048    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581050    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581053    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581056    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581059    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581061    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581064    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581067    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581069    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581072    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581074    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581077    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581080    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581083    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581085    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581088    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581090    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581093    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581096    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.656466 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581099    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581101    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581104    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581107    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581109    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581112    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581115    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581117    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581120    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581122    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581125    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581127    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581130    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581133    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581135    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581140    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581144    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581146    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581149    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.658757 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581152    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581154    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581157    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581159    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581162    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581164    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581167    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581170    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581172    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581175    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581177    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581179    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581182    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581185    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581187    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581190    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581193    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581195    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581198    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581200    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.660766 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581203    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581205    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581208    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581210    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581213    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581215    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581218    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581221    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581224    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581226    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581228    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581231    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581234    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581236    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581239    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581241    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581244    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581247    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581250    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581253    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.668704 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581255    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581260    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581263    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581266    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581269    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581272    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581274    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581277    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581281    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581284    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581287    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581290    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581293    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581295    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581298    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581661    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581666    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581669    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.669781 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581672    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581676    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581679    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581683    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581686    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581690    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581692    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581696    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581698    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581701    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581704    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581707    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581710    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581712    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581715    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581718    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581720    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581723    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581725    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.677293 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581728    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.673144 ip-10-0-129-47 systemd[1]: Started Kubernetes Kubelet.
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581730    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581733    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581735    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581738    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581741    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581744    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581747    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581749    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581751    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581754    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581757    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581759    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581762    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581765    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581768    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581770    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581773    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581775    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581778    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.678818 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581780    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581783    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581785    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581788    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581791    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581793    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581796    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581799    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581802    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581804    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581809    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581812    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581814    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581817    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581819    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581821    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581824    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581828    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581831    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581833    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.680024 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581836    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581838    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581841    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581843    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581846    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581848    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581851    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581853    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581856    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581858    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581861    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581863    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581866    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581869    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581872    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581875    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581877    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581880    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581882    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581885    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.681328 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581887    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581890    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581894    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581897    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581899    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581902    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581904    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581907    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581909    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581933    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581937    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581939    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581942    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581945    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581947    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581950    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.581953    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582035    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582046    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582055    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 09:16:55.682212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582062    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582067    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582072    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582079    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582084    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582087    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582090    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582093    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582097    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582100    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582103    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582106    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582108    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582112    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582117    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582121    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582124    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582127    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582130    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582133    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582137    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582141    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582144    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 09:16:55.683303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582147    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582150    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582153    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582156    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582159    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582162    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582166    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582169    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582172    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582175    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582178    2560 flags.go:64] FLAG: --enable-server="true"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582181    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582185    2560 flags.go:64] FLAG: --event-burst="100"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582188    2560 flags.go:64] FLAG: --event-qps="50"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582191    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582194    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582197    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582201    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582204    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582207    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582210    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582213    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582216    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582223    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582226    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 09:16:55.684413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582229    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582232    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582235    2560 flags.go:64] FLAG: --feature-gates=""
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582239    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582242    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582247    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582250    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582253    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582256    2560 flags.go:64] FLAG: --help="false"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582259    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-47.ec2.internal"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582262    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582265    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582268    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582271    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582275    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582278    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582281    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582283    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582286    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582289    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582293    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582296    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582299    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582301    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 09:16:55.685715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582304    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582307    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582310    2560 flags.go:64] FLAG: --lock-file=""
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582313    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582316    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582319    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582325    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582329    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582332    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582334    2560 flags.go:64] FLAG: --logging-format="text"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582337    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582341    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582343    2560 flags.go:64] FLAG: --manifest-url=""
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582350    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582355    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582358    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582362    2560 flags.go:64] FLAG: --max-pods="110"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582365    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582368    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582371    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582373    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582377    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582380    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582382    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582389    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 09:16:55.686769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582392    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582395    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582399    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582401    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582406    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582409    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582412    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582415    2560 flags.go:64] FLAG: --port="10250"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582418    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582421    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ac3635b7bc339aae"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582424    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582428    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582431    2560 flags.go:64] FLAG: --register-node="true"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582433    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582438    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582442    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582444    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582447    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582451    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582454    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582458    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582461    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582464    2560 flags.go:64] FLAG: --runonce="false"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582467    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582470    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 09:16:55.688294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582473    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582475    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582478    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582482    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582485    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582488    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582491    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582494    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582497    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582499    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582502    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582505    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582508    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582514    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582517    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582519    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582523    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582526    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582529    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582532    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582535    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582539    2560 flags.go:64] FLAG: --v="2"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582543    2560 flags.go:64] FLAG: --version="false"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582549    2560 flags.go:64] FLAG: --vmodule=""
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582552    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 09:16:55.689290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.582556    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582667    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582672    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582675    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582678    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582680    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582683    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582685    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582688    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582692    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582696    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582699    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582702    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582704    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582707    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582710    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582712    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582715    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.690338 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582729    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582733    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582736    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582739    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582742    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582744    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582747    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582750    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582752    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582755    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582759    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582761    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582764    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582767    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582770    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582773    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582776    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582779    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582782    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582784    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.691193 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582788    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582792    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582795    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582798    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582800    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582803    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582805    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582808    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582811    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582814    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582816    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582819    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582821    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582824    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582827    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582829    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582832    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582834    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582837    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582839    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.692054 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582842    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582845    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582849    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582852    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582854    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582857    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582860    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582862    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582866    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582869    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582871    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582874    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582876    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582879    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582882    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582884    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582887    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582889    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582892    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582895    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.692952 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582897    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582900    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582902    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582905    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582907    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582910    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582927    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582929    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582932    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582935    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582938    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582941    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582943    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582946    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582950    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582952    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582955    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582957    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582960    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582963    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.693823 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582967    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.582970    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.583589    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.590214    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.590229    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.591064    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.591071    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591131    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591137    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591141    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591144    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591147    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591150    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591154    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591158    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.694677 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591161    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591164    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591168    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591170    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591174    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591177    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591179    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591182    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591185    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591188    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591191    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591195    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591197    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591200    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591202    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591205    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591208    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591211    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591213    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.695522 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591216    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591219    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591221    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591224    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591226    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591229    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591233    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591236    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591240    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591243    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591246    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591249    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591251    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591254    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591259    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591261    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591264    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591267    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591269    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.696394 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591272    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591275    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591278    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591280    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591283    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591285    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591288    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591290    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591293    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591295    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591298    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591301    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591304    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591307    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591309    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591312    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591314    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591317    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591319    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591322    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.697257 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591324    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591327    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591329    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591332    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591334    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591337    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591339    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591343    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591348    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591350    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591353    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591356    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591358    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591361    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591364    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591366    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591369    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591371    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591374    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591377    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.700345 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591379    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591382    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591384    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591387    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591390    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591392    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591395    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591397    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591400    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591402    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591405    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591408    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591410    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.591415    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591523    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 09:16:55.701390 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591528    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591531    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591534    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591538    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591542    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591547    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591550    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591553    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591555    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591558    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591561    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591563    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591566    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591568    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591571    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591573    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591576    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591578    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591581    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 09:16:55.840121 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591583    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591586    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591588    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591591    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591594    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591596    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591599    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591601    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591604    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591606    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591609    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591612    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591614    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591617    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591619    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591622    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591624    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591627    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591632    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591634    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 09:16:55.863354 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591637    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591640    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591642    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591645    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591648    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591650    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591653    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591655    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591658    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591660    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591663    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591666    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591668    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591671    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591674    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591676    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591679    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591682    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591685    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 09:16:55.864097 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591689    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591691    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591694    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591696    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591699    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591701    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591704    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591707    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591709    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591712    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591714    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591722    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591725    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591728    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591732    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591735    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591737    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591740    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591743    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591746    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 09:16:55.864672 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591757    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591760    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591763    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591766    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591769    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591771    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591774    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591777    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591779    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591782    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591785    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591788    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591790    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591793    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591795    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591798    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591800    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591803    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591806    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 09:16:55.865777 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:55.591808    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.591813    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.591904    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.594009    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.595241    2560 server.go:1019] "Starting client certificate rotation"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.595342    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.596004    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.597504    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.601590    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.613481    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.626512    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.643203    2560 log.go:25] "Validated CRI v1 image API"
Jul 23 09:16:55.866336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.643993    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 09:16:55.866787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.647029    2560 fs.go:135] Filesystem UUIDs: map[01c2b361-3cc0-44cc-ad5f-00303ae7079b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b40424ae-cd42-4daf-915f-e385a5778931:/dev/nvme0n1p4]
Jul 23 09:16:55.866787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.647044    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.652194    2560 manager.go:216] Machine: {Timestamp:2026-07-23 09:16:55.651713238 +0000 UTC m=+0.329931032 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089173 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26681d2413931d3e9dcd293cd7ae9f SystemUUID:ec26681d-2413-931d-3e9d-cd293cd7ae9f BootID:da74fdee-8404-4a22-90cf-300a2f415243 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7c:da:f9:09:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7c:da:f9:09:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:bb:95:c9:df:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.652282    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.652342    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.655575    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.655601    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-47.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.656370    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.656382    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.656392    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.656409    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.657658    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661355    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661466    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661494    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661510    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661524    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661533    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661541    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 09:16:55.867021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661862    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.661876    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.665158    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.667156    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.667192    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668901    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668932    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668941    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668950    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668962    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668970    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668981    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668989    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.668998    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.669007    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.669019    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.669026    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.669687    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.669695    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.672547    2560 server.go:1268] "Started kubelet"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.672845    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.674873    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.676181    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.676224    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.679377    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.682734    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.683098    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.685300    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.685773    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.685791    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.685854    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.685955    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.685963    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686367    2560 factory.go:55] Registering systemd factory
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686384    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686589    2560 factory.go:153] Registering CRI-O factory
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686600    2560 factory.go:223] Registration of the crio container factory successfully
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686663    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686686    2560 factory.go:103] Registering Raw factory
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.686700    2560 manager.go:1196] Started watching for new ooms in manager
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.687110    2560 manager.go:349] Starting recovery of all containers
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.692950    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.715281    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-glf7c"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.715305    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-glf7c"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.715977    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.716455    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-47.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.716707    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 09:16:56.024513 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.717427    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-47.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.717432    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.717443    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.716529    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-47.ec2.internal.18c4e08558e7b378  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-47.ec2.internal,UID:ip-10-0-129-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-47.ec2.internal,},FirstTimestamp:2026-07-23 09:16:55.672411 +0000 UTC m=+0.350628773,LastTimestamp:2026-07-23 09:16:55.672411 +0000 UTC m=+0.350628773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-47.ec2.internal,}"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.718770    2560 manager.go:354] Recovery completed
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.723130    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.728552    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.728575    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.728586    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729036    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729044    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729044    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729076    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729661    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729681    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.729691    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.731013    2560 policy_none.go:49] "None policy: Start"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.731027    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.731040    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.731200    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-47.ec2.internal.18c4e0855c408298  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-47.ec2.internal,UID:ip-10-0-129-47.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-47.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-47.ec2.internal,},FirstTimestamp:2026-07-23 09:16:55.72856284 +0000 UTC m=+0.406780615,LastTimestamp:2026-07-23 09:16:55.72856284 +0000 UTC m=+0.406780615,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-47.ec2.internal,}"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.732959    2560 policy_none.go:47] "Start"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.779453    2560 manager.go:337] "Starting Device Plugin manager"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.779596    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.779610    2560 server.go:87] "Starting device plugin registration server"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.779932    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.779953    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.780054    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.780150    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.780159    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.780676    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.780714    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.880659    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.026101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.881359    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.881383    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.881392    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.881414    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.891104    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.891124    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-47.ec2.internal\": node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.905939    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.905966    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.905983    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.906018    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:55.908005    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:55.910326    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.006976    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"]
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.007054    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.010548    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.010894    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.010934    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.010944    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013146    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013307    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013332    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013961    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013979    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.013988    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.014304    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.014320    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.014328    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.015712    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.015732    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.017619    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.017637    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 09:16:56.027037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.017648    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeHasSufficientPID"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.058540    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-47.ec2.internal\" not found" node="ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.063536    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-47.ec2.internal\" not found" node="ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.087155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.087176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.087200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea26f9754e040fa3e04439c1b1bcf115-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"ea26f9754e040fa3e04439c1b1bcf115\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.111316    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea26f9754e040fa3e04439c1b1bcf115-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"ea26f9754e040fa3e04439c1b1bcf115\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/778ddef61fddd3352f7a606b7caeab8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal\" (UID: \"778ddef61fddd3352f7a606b7caeab8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.196943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.187546    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea26f9754e040fa3e04439c1b1bcf115-config\") pod \"kube-apiserver-proxy-ip-10-0-129-47.ec2.internal\" (UID: \"ea26f9754e040fa3e04439c1b1bcf115\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.323512 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.212230    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.323512 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.313642    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.479775 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.369254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.479775 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.369267    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:56.479775 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.414401    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.624267 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.515469    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.624267 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.594776    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 09:16:56.624267 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.594961    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 09:16:56.624267 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.616427    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.787602 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.716632    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.787602 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.720303    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 09:11:56 +0000 UTC" deadline="2028-02-14 12:42:57.383822433 +0000 UTC"
Jul 23 09:16:56.787602 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.720336    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13707h26m0.663489782s"
Jul 23 09:16:56.787602 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.724927    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 09:16:56.918656 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.817122    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:56.918656 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.912901    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:56.936969    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-47.ec2.internal\" not found"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.937240    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:56.985873    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.003156    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.004038    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.013538    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:57.028188    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea26f9754e040fa3e04439c1b1bcf115.slice/crio-8508e3bfafd6b6b5e962817c55a5113418d509ecb2f5b02cd814b2278e18996b WatchSource:0}: Error finding container 8508e3bfafd6b6b5e962817c55a5113418d509ecb2f5b02cd814b2278e18996b: Status 404 returned error can't find the container with id 8508e3bfafd6b6b5e962817c55a5113418d509ecb2f5b02cd814b2278e18996b
Jul 23 09:16:57.086609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.033560    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:16:57.665894 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.665653    2560 apiserver.go:52] "Watching apiserver"
Jul 23 09:16:57.675383 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.675196    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 09:16:57.677242 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.677206    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-q7lc8","openshift-multus/multus-additional-cni-plugins-rl4q7","openshift-multus/network-metrics-daemon-hk2bf","openshift-image-registry/node-ca-xzzhs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal","openshift-multus/multus-2l9bs","openshift-network-diagnostics/network-check-target-nmlww","openshift-network-operator/iptables-alerter-h54vv","openshift-ovn-kubernetes/ovnkube-node-mnc5x","kube-system/global-pull-secret-syncer-9rq4z","kube-system/konnectivity-agent-czvh8","kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw","openshift-cluster-node-tuning-operator/tuned-jg7bn"]
Jul 23 09:16:57.678741 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:57.678715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778ddef61fddd3352f7a606b7caeab8d.slice/crio-38b216efe5474449ee5dcd96564fd1bdb55b0448578ec9b4fd69f472b07dca0b WatchSource:0}: Error finding container 38b216efe5474449ee5dcd96564fd1bdb55b0448578ec9b4fd69f472b07dca0b: Status 404 returned error can't find the container with id 38b216efe5474449ee5dcd96564fd1bdb55b0448578ec9b4fd69f472b07dca0b
Jul 23 09:16:57.680406 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.680384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:57.680504 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.680466    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:16:57.683109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.682588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.685101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.685076    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.685222 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.685157    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.685943    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.686429    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zjsk6\""
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.686614    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.686644    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.686744    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 09:16:57.686841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.686750    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.687320 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.687189    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 09:16:57.687486 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.687467    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.694046 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.693737    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 09:16:57.694046 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.693743    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vx2ms\""
Jul 23 09:16:57.694304 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.694281    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.694304 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.694294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.694439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.694424    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.696557 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.696534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.698408 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xpl\" (UniqueName: \"kubernetes.io/projected/f744bf95-832b-4e38-8967-3f794694cdbd-kube-api-access-d7xpl\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.698532 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:57.698532 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.698655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f744bf95-832b-4e38-8967-3f794694cdbd-serviceca\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.698655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-system-cni-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-cnibin\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-os-release\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5ch\" (UniqueName: \"kubernetes.io/projected/4797eef2-0e43-426d-bbfb-70b818f55b01-kube-api-access-5w5ch\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.698805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.699132 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz82w\" (UniqueName: \"kubernetes.io/projected/460e979e-3746-4b38-8768-779560aba737-kube-api-access-wz82w\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.699132 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.698846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f744bf95-832b-4e38-8967-3f794694cdbd-host\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.699240 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.699183    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.699338 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.699317    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r92zm\""
Jul 23 09:16:57.699404 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.699368    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-974zc\""
Jul 23 09:16:57.699463 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.699414    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 09:16:57.699651 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.699608    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.700292 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.700259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.702612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.702380    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 09:16:57.702710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.702686    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xcctd\""
Jul 23 09:16:57.703163 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.703131    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.703263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.703196    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 09:16:57.703263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.703232    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.704556 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.703804    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.706078 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.706057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8pqxc\""
Jul 23 09:16:57.706078 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.706077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.706281 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.706132    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:16:57.706375 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.706354    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 09:16:57.706575 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.706354    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 09:16:57.708729 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.708657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.711775 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.711754    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.712498 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.712480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.712636 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.712531    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 09:16:57.712887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.712866    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdb2j\""
Jul 23 09:16:57.713860 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.713840    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.714540 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.714520    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.714638 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.714546    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k8hnw\""
Jul 23 09:16:57.714638 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.714626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.715681 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.715598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 09:16:57.715681 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.715648    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 09:16:57.715681 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.715664    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 09:16:57.715873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.715652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 09:16:57.715873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.715598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.718266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.718136    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 09:16:57.718266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.718164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p46nh\""
Jul 23 09:16:57.718266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.718206    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 09:16:57.720736 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.720717    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 09:16:57.727386 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.727364    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-42j6z"
Jul 23 09:16:57.736823 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.736804    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-42j6z"
Jul 23 09:16:57.787473 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.787453    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 09:16:57.800091 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-script-lib\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-host\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-dbus\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-systemd\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sj4\" (UniqueName: \"kubernetes.io/projected/b23288ec-5ac5-475c-8e79-86dfab6df91b-kube-api-access-m7sj4\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800254    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-os-release\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5ch\" (UniqueName: \"kubernetes.io/projected/4797eef2-0e43-426d-bbfb-70b818f55b01-kube-api-access-5w5ch\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.800291    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800351    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-hostroot\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.800375    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:58.300351448 +0000 UTC m=+2.978569210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800380    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-os-release\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-sys-fs\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800578    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-systemd-units\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-modprobe-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-run\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.800673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-os-release\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcdw\" (UniqueName: \"kubernetes.io/projected/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-kube-api-access-mhcdw\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-log-socket\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-sys\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-daemon-config\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04d9c8af-c8e3-4066-980a-479d1d167adc-iptables-alerter-script\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04d9c8af-c8e3-4066-980a-479d1d167adc-host-slash\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsk6\" (UniqueName: \"kubernetes.io/projected/d72157fc-26b6-4c02-b1fc-8631f6a9d570-kube-api-access-4dsk6\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xpl\" (UniqueName: \"kubernetes.io/projected/f744bf95-832b-4e38-8967-3f794694cdbd-kube-api-access-d7xpl\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.800998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cni-binary-copy\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f744bf95-832b-4e38-8967-3f794694cdbd-serviceca\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-multus-certs\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801074    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-kubelet\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-env-overrides\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-conf\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-tmp\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.801538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-kubelet\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-conf-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-k8s-cni-cncf-io\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2228873f-e114-44b8-80bf-34a2f2fc7932-agent-certs\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-etc-kubernetes\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67vr\" (UniqueName: \"kubernetes.io/projected/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-kube-api-access-x67vr\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-device-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f744bf95-832b-4e38-8967-3f794694cdbd-serviceca\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-systemd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-node-log\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-netd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovn-node-metrics-cert\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-tuned\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-system-cni-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-bin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.802230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-system-cni-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801759    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-kubelet-config\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-hosts-file\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2228873f-e114-44b8-80bf-34a2f2fc7932-konnectivity-ca\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-etc-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-binary-copy\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.801968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-socket-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-registration-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-etc-selinux\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-config\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysconfig\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-kubernetes\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-var-lib-kubelet\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.803082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz82w\" (UniqueName: \"kubernetes.io/projected/460e979e-3746-4b38-8768-779560aba737-kube-api-access-wz82w\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-ovn\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-lib-modules\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f744bf95-832b-4e38-8967-3f794694cdbd-host\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f744bf95-832b-4e38-8967-3f794694cdbd-host\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-tmp-dir\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-var-lib-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-cnibin\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cnibin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-socket-dir-parent\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-bin\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.803593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/460e979e-3746-4b38-8768-779560aba737-cnibin\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-netns\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5z8\" (UniqueName: \"kubernetes.io/projected/540dec51-8eb1-4589-b040-979249b32436-kube-api-access-dc5z8\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-netns\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-system-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-multus\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/04d9c8af-c8e3-4066-980a-479d1d167adc-kube-api-access-hqmf8\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.802927    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-slash\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.804188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.803126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/460e979e-3746-4b38-8768-779560aba737-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.812091 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.812071    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:16:57.812200 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.812100    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:16:57.812200 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.812117    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:57.812200 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.812177    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:16:58.312160759 +0000 UTC m=+2.990378528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:57.812400 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.812379    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 09:16:57.816390 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.816369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz82w\" (UniqueName: \"kubernetes.io/projected/460e979e-3746-4b38-8768-779560aba737-kube-api-access-wz82w\") pod \"multus-additional-cni-plugins-rl4q7\" (UID: \"460e979e-3746-4b38-8768-779560aba737\") " pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:57.816497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.816390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5ch\" (UniqueName: \"kubernetes.io/projected/4797eef2-0e43-426d-bbfb-70b818f55b01-kube-api-access-5w5ch\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:57.816613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.816586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xpl\" (UniqueName: \"kubernetes.io/projected/f744bf95-832b-4e38-8967-3f794694cdbd-kube-api-access-d7xpl\") pod \"node-ca-xzzhs\" (UID: \"f744bf95-832b-4e38-8967-3f794694cdbd\") " pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:57.903325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-multus-certs\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.903438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-kubelet\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.903438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-env-overrides\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.903438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-conf\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-kubelet\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-multus-certs\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-conf\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-tmp\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-kubelet\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-conf-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-k8s-cni-cncf-io\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2228873f-e114-44b8-80bf-34a2f2fc7932-agent-certs\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903651    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-kubelet\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-conf-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.903676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-etc-kubernetes\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-k8s-cni-cncf-io\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-etc-kubernetes\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67vr\" (UniqueName: \"kubernetes.io/projected/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-kube-api-access-x67vr\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysctl-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-device-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-systemd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-device-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-env-overrides\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-node-log\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903894    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-node-log\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-systemd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-netd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovn-node-metrics-cert\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.903991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-tuned\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-netd\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904100    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-bin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-kubelet-config\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.904312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-hosts-file\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-bin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2228873f-e114-44b8-80bf-34a2f2fc7932-konnectivity-ca\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-etc-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-kubelet-config\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-hosts-file\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-etc-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-socket-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-registration-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-etc-selinux\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-config\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-registration-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.905254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904524    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysconfig\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-kubernetes\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-sysconfig\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-var-lib-kubelet\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-ovn\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-lib-modules\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-etc-selinux\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-tmp-dir\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-ovn-kubernetes\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-kubernetes\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2228873f-e114-44b8-80bf-34a2f2fc7932-konnectivity-ca\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904831    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-var-lib-kubelet\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-var-lib-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cnibin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-run-ovn\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-socket-dir-parent\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-bin\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.906125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.904951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-netns\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5z8\" (UniqueName: \"kubernetes.io/projected/540dec51-8eb1-4589-b040-979249b32436-kube-api-access-dc5z8\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-netns\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-system-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-lib-modules\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-multus\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/04d9c8af-c8e3-4066-980a-479d1d167adc-kube-api-access-hqmf8\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-slash\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.905166    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-config\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:57.905232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:58.405213002 +0000 UTC m=+3.083430771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-socket-dir\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-tmp-dir\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-script-lib\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905724    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovnkube-script-lib\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-host\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-var-lib-openvswitch\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cnibin\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-run-netns\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.905963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-socket-dir-parent\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-host\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-system-cni-dir\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-var-lib-cni-multus\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-tmp\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-slash\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-dbus\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-systemd\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-host-cni-bin\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sj4\" (UniqueName: \"kubernetes.io/projected/b23288ec-5ac5-475c-8e79-86dfab6df91b-kube-api-access-m7sj4\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-hostroot\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-host-run-netns\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-sys-fs\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-systemd-units\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.907724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/540dec51-8eb1-4589-b040-979249b32436-sys-fs\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-modprobe-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-run\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-dbus\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-os-release\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcdw\" (UniqueName: \"kubernetes.io/projected/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-kube-api-access-mhcdw\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-systemd\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-log-socket\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-sys\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-daemon-config\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-modprobe-d\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04d9c8af-c8e3-4066-980a-479d1d167adc-iptables-alerter-script\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04d9c8af-c8e3-4066-980a-479d1d167adc-host-slash\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsk6\" (UniqueName: \"kubernetes.io/projected/d72157fc-26b6-4c02-b1fc-8631f6a9d570-kube-api-access-4dsk6\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-os-release\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cni-binary-copy\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-run\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/04d9c8af-c8e3-4066-980a-479d1d167adc-host-slash\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.908362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-systemd-units\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-hostroot\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23288ec-5ac5-475c-8e79-86dfab6df91b-sys\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2228873f-e114-44b8-80bf-34a2f2fc7932-agent-certs\") pod \"konnectivity-agent-czvh8\" (UID: \"2228873f-e114-44b8-80bf-34a2f2fc7932\") " pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.906879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d72157fc-26b6-4c02-b1fc-8631f6a9d570-log-socket\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.907333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d72157fc-26b6-4c02-b1fc-8631f6a9d570-ovn-node-metrics-cert\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.907412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-cni-binary-copy\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.907477    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/04d9c8af-c8e3-4066-980a-479d1d167adc-iptables-alerter-script\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.907513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-multus-daemon-config\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.909087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.907601    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b23288ec-5ac5-475c-8e79-86dfab6df91b-etc-tuned\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.910452 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.910409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"778ddef61fddd3352f7a606b7caeab8d","Type":"ContainerStarted","Data":"38b216efe5474449ee5dcd96564fd1bdb55b0448578ec9b4fd69f472b07dca0b"}
Jul 23 09:16:57.912042 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.912016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" event={"ID":"ea26f9754e040fa3e04439c1b1bcf115","Type":"ContainerStarted","Data":"8508e3bfafd6b6b5e962817c55a5113418d509ecb2f5b02cd814b2278e18996b"}
Jul 23 09:16:57.929997 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.929934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sj4\" (UniqueName: \"kubernetes.io/projected/b23288ec-5ac5-475c-8e79-86dfab6df91b-kube-api-access-m7sj4\") pod \"tuned-jg7bn\" (UID: \"b23288ec-5ac5-475c-8e79-86dfab6df91b\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:57.931307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.931263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmf8\" (UniqueName: \"kubernetes.io/projected/04d9c8af-c8e3-4066-980a-479d1d167adc-kube-api-access-hqmf8\") pod \"iptables-alerter-h54vv\" (UID: \"04d9c8af-c8e3-4066-980a-479d1d167adc\") " pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:57.932511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.932492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67vr\" (UniqueName: \"kubernetes.io/projected/6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300-kube-api-access-x67vr\") pod \"multus-2l9bs\" (UID: \"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300\") " pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:57.932789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.932770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5z8\" (UniqueName: \"kubernetes.io/projected/540dec51-8eb1-4589-b040-979249b32436-kube-api-access-dc5z8\") pod \"aws-ebs-csi-driver-node-9b2xw\" (UID: \"540dec51-8eb1-4589-b040-979249b32436\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:57.933303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.933279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcdw\" (UniqueName: \"kubernetes.io/projected/e6e5add7-e2a7-47a6-a2e9-8685a6af4dff-kube-api-access-mhcdw\") pod \"node-resolver-q7lc8\" (UID: \"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff\") " pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:57.934122 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.934099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsk6\" (UniqueName: \"kubernetes.io/projected/d72157fc-26b6-4c02-b1fc-8631f6a9d570-kube-api-access-4dsk6\") pod \"ovnkube-node-mnc5x\" (UID: \"d72157fc-26b6-4c02-b1fc-8631f6a9d570\") " pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:57.997324 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:57.997301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rl4q7"
Jul 23 09:16:58.007651 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.007621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xzzhs"
Jul 23 09:16:58.018518 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.018500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7lc8"
Jul 23 09:16:58.023155 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.023138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2l9bs"
Jul 23 09:16:58.031982 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.031955    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h54vv"
Jul 23 09:16:58.038737 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.038704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:16:58.045659 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.045637    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw"
Jul 23 09:16:58.053609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.053586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:16:58.060347 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.060328    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jg7bn"
Jul 23 09:16:58.309962 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.309881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:58.310110 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.310049    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:58.310176 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.310120    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:59.310101445 +0000 UTC m=+3.988319212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:58.410653 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.410612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:58.410833 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.410668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:58.410833 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410761    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:16:58.410833 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410787    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:16:58.410833 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410802    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:58.410833 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410800    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:58.411152 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410864    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:16:59.410843813 +0000 UTC m=+4.089061597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:58.411152 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.410884    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:16:59.410873937 +0000 UTC m=+4.089091703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:58.650029 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.649994    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744bf95_832b_4e38_8967_3f794694cdbd.slice/crio-8301dd1b34e0dacb710d2fcc3123b07cc02dde03f1b144f77ee5aaf7ad2cd7bb WatchSource:0}: Error finding container 8301dd1b34e0dacb710d2fcc3123b07cc02dde03f1b144f77ee5aaf7ad2cd7bb: Status 404 returned error can't find the container with id 8301dd1b34e0dacb710d2fcc3123b07cc02dde03f1b144f77ee5aaf7ad2cd7bb
Jul 23 09:16:58.672177 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.672150    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460e979e_3746_4b38_8768_779560aba737.slice/crio-429a8f44bd10d5b0cddd87215fd6d736ac833161c6b1d01d27d88af2f7271a79 WatchSource:0}: Error finding container 429a8f44bd10d5b0cddd87215fd6d736ac833161c6b1d01d27d88af2f7271a79: Status 404 returned error can't find the container with id 429a8f44bd10d5b0cddd87215fd6d736ac833161c6b1d01d27d88af2f7271a79
Jul 23 09:16:58.673597 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.673570    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e5add7_e2a7_47a6_a2e9_8685a6af4dff.slice/crio-252d3d11c6bbb9844c0dc0e87dcdd7b4772c29c93583b590c60cf4e90f004e2c WatchSource:0}: Error finding container 252d3d11c6bbb9844c0dc0e87dcdd7b4772c29c93583b590c60cf4e90f004e2c: Status 404 returned error can't find the container with id 252d3d11c6bbb9844c0dc0e87dcdd7b4772c29c93583b590c60cf4e90f004e2c
Jul 23 09:16:58.680171 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.680144    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6656f6a5_1eb8_4d38_8ee7_9a57e4fb4300.slice/crio-6b8538039b355c59fada752bfa60729e0ee5b829b557cfe3bda3078bae2cbef2 WatchSource:0}: Error finding container 6b8538039b355c59fada752bfa60729e0ee5b829b557cfe3bda3078bae2cbef2: Status 404 returned error can't find the container with id 6b8538039b355c59fada752bfa60729e0ee5b829b557cfe3bda3078bae2cbef2
Jul 23 09:16:58.681255 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.681230    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d9c8af_c8e3_4066_980a_479d1d167adc.slice/crio-ce3fc4945d7f74a3aff2fc32648d4ba20e2a7bfac3ad724f2f6c2c1fd629e3dc WatchSource:0}: Error finding container ce3fc4945d7f74a3aff2fc32648d4ba20e2a7bfac3ad724f2f6c2c1fd629e3dc: Status 404 returned error can't find the container with id ce3fc4945d7f74a3aff2fc32648d4ba20e2a7bfac3ad724f2f6c2c1fd629e3dc
Jul 23 09:16:58.683717 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.683673    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72157fc_26b6_4c02_b1fc_8631f6a9d570.slice/crio-9dea222d63850ebf49b932433d95f782dd7ce62e571f261f370169e79ab37e14 WatchSource:0}: Error finding container 9dea222d63850ebf49b932433d95f782dd7ce62e571f261f370169e79ab37e14: Status 404 returned error can't find the container with id 9dea222d63850ebf49b932433d95f782dd7ce62e571f261f370169e79ab37e14
Jul 23 09:16:58.685531 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.685327    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23288ec_5ac5_475c_8e79_86dfab6df91b.slice/crio-97647275f7ee08202566c2329d8058bda069e3a8a62ef596fce5b9a9af4f1d26 WatchSource:0}: Error finding container 97647275f7ee08202566c2329d8058bda069e3a8a62ef596fce5b9a9af4f1d26: Status 404 returned error can't find the container with id 97647275f7ee08202566c2329d8058bda069e3a8a62ef596fce5b9a9af4f1d26
Jul 23 09:16:58.686451 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.686426    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2228873f_e114_44b8_80bf_34a2f2fc7932.slice/crio-401435f449cc39442d19c9c9b31c53a7def31efbc77f2f05c7cd5dbb7c6faf48 WatchSource:0}: Error finding container 401435f449cc39442d19c9c9b31c53a7def31efbc77f2f05c7cd5dbb7c6faf48: Status 404 returned error can't find the container with id 401435f449cc39442d19c9c9b31c53a7def31efbc77f2f05c7cd5dbb7c6faf48
Jul 23 09:16:58.688950 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:16:58.688761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540dec51_8eb1_4589_b040_979249b32436.slice/crio-4ff601a49d40c6323b04aad2203ef7be1e9f30f8c597b72d52102f22099ca5a2 WatchSource:0}: Error finding container 4ff601a49d40c6323b04aad2203ef7be1e9f30f8c597b72d52102f22099ca5a2: Status 404 returned error can't find the container with id 4ff601a49d40c6323b04aad2203ef7be1e9f30f8c597b72d52102f22099ca5a2
Jul 23 09:16:58.740518 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.740466    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 09:11:58 +0000 UTC" deadline="2028-02-23 21:25:33.877339884 +0000 UTC"
Jul 23 09:16:58.740613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.740519    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13932h8m35.136824482s"
Jul 23 09:16:58.906458 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.906384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:58.907165 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:58.906500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:16:58.916839 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.916804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" event={"ID":"ea26f9754e040fa3e04439c1b1bcf115","Type":"ContainerStarted","Data":"5f451fd98fa509fe87cfe3f5e9155b407b397be4f6c52816b19b6470a0c9dc5b"}
Jul 23 09:16:58.918123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.918092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw" event={"ID":"540dec51-8eb1-4589-b040-979249b32436","Type":"ContainerStarted","Data":"4ff601a49d40c6323b04aad2203ef7be1e9f30f8c597b72d52102f22099ca5a2"}
Jul 23 09:16:58.919363 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.919338    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h54vv" event={"ID":"04d9c8af-c8e3-4066-980a-479d1d167adc","Type":"ContainerStarted","Data":"ce3fc4945d7f74a3aff2fc32648d4ba20e2a7bfac3ad724f2f6c2c1fd629e3dc"}
Jul 23 09:16:58.920640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.920617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerStarted","Data":"429a8f44bd10d5b0cddd87215fd6d736ac833161c6b1d01d27d88af2f7271a79"}
Jul 23 09:16:58.921885 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.921854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-czvh8" event={"ID":"2228873f-e114-44b8-80bf-34a2f2fc7932","Type":"ContainerStarted","Data":"401435f449cc39442d19c9c9b31c53a7def31efbc77f2f05c7cd5dbb7c6faf48"}
Jul 23 09:16:58.923166 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.923144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jg7bn" event={"ID":"b23288ec-5ac5-475c-8e79-86dfab6df91b","Type":"ContainerStarted","Data":"97647275f7ee08202566c2329d8058bda069e3a8a62ef596fce5b9a9af4f1d26"}
Jul 23 09:16:58.924343 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.924311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"9dea222d63850ebf49b932433d95f782dd7ce62e571f261f370169e79ab37e14"}
Jul 23 09:16:58.925557 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.925539    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l9bs" event={"ID":"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300","Type":"ContainerStarted","Data":"6b8538039b355c59fada752bfa60729e0ee5b829b557cfe3bda3078bae2cbef2"}
Jul 23 09:16:58.926726 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.926706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7lc8" event={"ID":"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff","Type":"ContainerStarted","Data":"252d3d11c6bbb9844c0dc0e87dcdd7b4772c29c93583b590c60cf4e90f004e2c"}
Jul 23 09:16:58.927885 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.927863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xzzhs" event={"ID":"f744bf95-832b-4e38-8967-3f794694cdbd","Type":"ContainerStarted","Data":"8301dd1b34e0dacb710d2fcc3123b07cc02dde03f1b144f77ee5aaf7ad2cd7bb"}
Jul 23 09:16:58.936135 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:58.936082    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-47.ec2.internal" podStartSLOduration=1.9360665190000002 podStartE2EDuration="1.936066519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:16:58.935941589 +0000 UTC m=+3.614159398" watchObservedRunningTime="2026-07-23 09:16:58.936066519 +0000 UTC m=+3.614284305"
Jul 23 09:16:59.317566 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.317426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:59.317732 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.317589    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:59.317732 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.317659    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.317639829 +0000 UTC m=+5.995857612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:16:59.419255 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.419187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:16:59.419440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.419272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:59.419440 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.419415    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:59.419550 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.419508    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.419488246 +0000 UTC m=+6.097706024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:16:59.420074 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.420031    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:16:59.420074 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.420059    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:16:59.420074 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.420072    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:59.420328 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.420128    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:17:01.420112849 +0000 UTC m=+6.098330618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:16:59.741118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.740936    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 09:11:58 +0000 UTC" deadline="2028-04-09 17:56:15.792383596 +0000 UTC"
Jul 23 09:16:59.741118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.740983    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15032h39m16.051405795s"
Jul 23 09:16:59.920108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.919117    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:16:59.920108 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.919312    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:16:59.920108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.919862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:16:59.920108 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:16:59.920032    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:16:59.955778 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.955186    2560 generic.go:356] "Generic (PLEG): container finished" podID="778ddef61fddd3352f7a606b7caeab8d" containerID="0458414f5e6c24249315996f797c90e96674290dda1fc395ffb832f7c81b02b3" exitCode=0
Jul 23 09:16:59.955778 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:16:59.955703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"778ddef61fddd3352f7a606b7caeab8d","Type":"ContainerDied","Data":"0458414f5e6c24249315996f797c90e96674290dda1fc395ffb832f7c81b02b3"}
Jul 23 09:17:00.906390 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:00.906344    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:00.906574 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:00.906472    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:00.990017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:00.989920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" event={"ID":"778ddef61fddd3352f7a606b7caeab8d","Type":"ContainerStarted","Data":"aa671052ae6e7b8c43f10b90deb1aa8240b7dd659529cf52971fde679b70e072"}
Jul 23 09:17:01.013948 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.013876    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-47.ec2.internal" podStartSLOduration=4.013854215 podStartE2EDuration="4.013854215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:01.012654365 +0000 UTC m=+5.690872156" watchObservedRunningTime="2026-07-23 09:17:01.013854215 +0000 UTC m=+5.692072007"
Jul 23 09:17:01.336297 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.336261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:01.336602 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.336531    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:01.336695 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.336682    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:05.336660622 +0000 UTC m=+10.014878388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:01.437673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.437630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:01.437879 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.437687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:01.437978 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.437879    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:01.437978 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.437942    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:05.437924002 +0000 UTC m=+10.116141771 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:01.438115 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.438060    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:17:01.438115 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.438076    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:17:01.438115 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.438088    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:01.438268 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.438125    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:17:05.438112599 +0000 UTC m=+10.116330375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:01.909718 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.907190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:01.909718 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.907338    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:01.909718 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:01.907203    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:01.909718 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:01.907849    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:02.907713 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:02.907139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:02.907713 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:02.907273    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:03.908613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:03.907849    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:03.908613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:03.907897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:03.908613 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:03.908085    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:03.908613 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:03.908239    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:04.906291 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:04.906251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:04.906455 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:04.906387    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:05.371568 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:05.371527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:05.372072 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.371698    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:05.372072 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.371770    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:13.371750582 +0000 UTC m=+18.049968366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:05.472633 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:05.472593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:05.472633 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:05.472649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:05.472887 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.472813    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:05.472967 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.472890    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:13.472858766 +0000 UTC m=+18.151076550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:05.473219 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.473195    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:17:05.473219 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.473222    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:17:05.473405 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.473236    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:05.473405 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.473294    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:17:13.473278367 +0000 UTC m=+18.151496135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:05.907368 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:05.907171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:05.907368 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.907312    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:05.907748 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:05.907688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:05.907886 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:05.907795    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:06.906428 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:06.906391    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:06.906925 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:06.906539    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:07.906751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:07.906484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:07.906751 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:07.906617    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:07.907263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:07.907054    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:07.907263 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:07.907171    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:08.907053 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:08.907018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:08.907547 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:08.907149    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:09.906427 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:09.906397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:09.906583 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:09.906500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:09.906625 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:09.906598    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:09.906744 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:09.906721    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:10.907202 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:10.907170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:10.907645 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:10.907296    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:11.906519 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:11.906484    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:11.906732 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:11.906618    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:11.906732 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:11.906677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:11.906854 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:11.906794    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:12.906348 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:12.906315    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:12.906826 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:12.906462    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:13.436590 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:13.436544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:13.436758 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.436709    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:13.436840 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.436795    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.436780107 +0000 UTC m=+34.114997873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:13.537681 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:13.537643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:13.537853 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:13.537694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:13.537853 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537813    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:13.537853 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537815    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:17:13.537853 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537836    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:17:13.537853 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537848    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:13.538077 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537885    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.53786678 +0000 UTC m=+34.216084551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:13.538077 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.537904    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:17:29.537893588 +0000 UTC m=+34.216111355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:13.907001 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:13.906900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:13.907433 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:13.906906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:13.907433 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.907040    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:13.907433 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:13.907125    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:14.906846 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:14.906812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:14.907035 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:14.906946    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:15.907440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:15.907401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:15.907889 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:15.907518    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:15.907889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:15.907579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:15.907889 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:15.907698    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:16.906477 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:16.906443    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:16.906676 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:16.906573    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:17.906479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:17.906444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:17.906999 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:17.906561    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:17.906999 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:17.906650    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:17.906999 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:17.906780    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:18.907127 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:18.906952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:18.907722 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:18.907210    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:19.026196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.026127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jg7bn" event={"ID":"b23288ec-5ac5-475c-8e79-86dfab6df91b","Type":"ContainerStarted","Data":"86e220158710079cb11e4c65861dad0b43136a131f4041308c51acc100035934"}
Jul 23 09:17:19.033053 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"88ab083dffe5d55775d286a49fc5e0b5734f36357e1149e5ca2e84f674968741"}
Jul 23 09:17:19.033202 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"7fa5ef45b07e4dd71a801fa414bea892e004ecb629de03ce31c51984849f6728"}
Jul 23 09:17:19.033202 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"3770b7405414cde533b26a49b9fde7873daca0a1600c8df001502f77e89ba1ec"}
Jul 23 09:17:19.033307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"65a48785269f02e1169fec343e93475b88630740efe764dd6432cffa595cc65d"}
Jul 23 09:17:19.033345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"b51c2a37907ae2ced6ec928d9cb0849e013258e0fe08ccf8348c6b018eb52b7a"}
Jul 23 09:17:19.033387 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.033370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"54e4921eced08f3f92340b99ff783d511e0fff6e42a201b9a362328eec9ae39e"}
Jul 23 09:17:19.034647 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.034625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2l9bs" event={"ID":"6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300","Type":"ContainerStarted","Data":"eb210fb147e2b593b3685483073caed22a073ff990e91cf599272b0d530952ee"}
Jul 23 09:17:19.036074 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.036049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7lc8" event={"ID":"e6e5add7-e2a7-47a6-a2e9-8685a6af4dff","Type":"ContainerStarted","Data":"40b5ca74b39055ade53c0151093ee3007d559b24ca389d9b9594cc1a6d86a797"}
Jul 23 09:17:19.037449 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.037428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xzzhs" event={"ID":"f744bf95-832b-4e38-8967-3f794694cdbd","Type":"ContainerStarted","Data":"9356a250e3a5baa9aa9756269852415dc25974320575b521a39f739dd4f25a31"}
Jul 23 09:17:19.038800 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.038778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw" event={"ID":"540dec51-8eb1-4589-b040-979249b32436","Type":"ContainerStarted","Data":"ec3ad526f05111849a3a3228ae209aae92605a464782a0161c792ee85e581094"}
Jul 23 09:17:19.040362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.040327    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="e9b15b3393af5b3ad8eab733d998d7a20992f0ea467723365ba0739e6d73b472" exitCode=0
Jul 23 09:17:19.040452 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.040381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"e9b15b3393af5b3ad8eab733d998d7a20992f0ea467723365ba0739e6d73b472"}
Jul 23 09:17:19.042031 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.042009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-czvh8" event={"ID":"2228873f-e114-44b8-80bf-34a2f2fc7932","Type":"ContainerStarted","Data":"59e5a1b2464e9de90adbb63396f1eebeae12f2e8b4163ad18fdd897acf96c14d"}
Jul 23 09:17:19.045804 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.045761    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jg7bn" podStartSLOduration=2.544095038 podStartE2EDuration="22.04575084s" totalImagesPullingTime="19.501655802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:19.04569339 +0000 UTC m=+23.723911173" watchObservedRunningTime="2026-07-23 09:17:19.04575084 +0000 UTC m=+23.723968622"
Jul 23 09:17:19.062591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.062557    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-czvh8" podStartSLOduration=11.581671417 podStartE2EDuration="23.062545473s" totalImagesPullingTime="11.480874056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:19.061949441 +0000 UTC m=+23.740167253" watchObservedRunningTime="2026-07-23 09:17:19.062545473 +0000 UTC m=+23.740763257"
Jul 23 09:17:19.077999 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.077946    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xzzhs" podStartSLOduration=3.557492429 podStartE2EDuration="23.077934446s" totalImagesPullingTime="19.520442017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:19.077720598 +0000 UTC m=+23.755938381" watchObservedRunningTime="2026-07-23 09:17:19.077934446 +0000 UTC m=+23.756152229"
Jul 23 09:17:19.092898 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.092867    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q7lc8" podStartSLOduration=3.579241177 podStartE2EDuration="23.092859356s" totalImagesPullingTime="19.513618179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:19.092431904 +0000 UTC m=+23.770649687" watchObservedRunningTime="2026-07-23 09:17:19.092859356 +0000 UTC m=+23.771077138"
Jul 23 09:17:19.133225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.133191    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2l9bs" podStartSLOduration=3.572541683 podStartE2EDuration="23.133180251s" totalImagesPullingTime="19.560638568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:19.132873963 +0000 UTC m=+23.811091776" watchObservedRunningTime="2026-07-23 09:17:19.133180251 +0000 UTC m=+23.811398033"
Jul 23 09:17:19.492543 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.492517    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 09:17:19.796221 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.796078    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T09:17:19.49253763Z","UUID":"f230fb58-287c-4c6f-8b20-18ac21d06d2d","Handler":null,"Name":"","Endpoint":""}
Jul 23 09:17:19.797779 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.797750    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 09:17:19.797898 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.797791    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 09:17:19.907056 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.907026    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:19.907290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:19.907058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:19.907290 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:19.907156    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:19.907774 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:19.907302    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:20.046198 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.046165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw" event={"ID":"540dec51-8eb1-4589-b040-979249b32436","Type":"ContainerStarted","Data":"e85c186e022f3df1f2765aa371f6cafd06a126033590df8c47fe14a955cc3854"}
Jul 23 09:17:20.048071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.047986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h54vv" event={"ID":"04d9c8af-c8e3-4066-980a-479d1d167adc","Type":"ContainerStarted","Data":"c6d946ad8caaf1357db6c6afdaf2edcd1ff5c87b9a31ba8c32c323578e412d01"}
Jul 23 09:17:20.123825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.123790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:17:20.124399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.124379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:17:20.140874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.140837    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h54vv" podStartSLOduration=4.63561673 podStartE2EDuration="24.140823231s" totalImagesPullingTime="19.505206501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:20.065226553 +0000 UTC m=+24.743444342" watchObservedRunningTime="2026-07-23 09:17:20.140823231 +0000 UTC m=+24.819041044"
Jul 23 09:17:20.906874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:20.906848    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:20.907007 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:20.906963    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:21.054196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.054164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"acc157c7eafc8f7db8dbcfd36dbffbaf018951cf9776e5aac1cd70a751bca43d"}
Jul 23 09:17:21.056651 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.056622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw" event={"ID":"540dec51-8eb1-4589-b040-979249b32436","Type":"ContainerStarted","Data":"b36a17861f5f6f0ff99b535c8bf660a1ce30bbbfc408eb263b33b1a3941ab416"}
Jul 23 09:17:21.081640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.081593    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9b2xw" podStartSLOduration=2.440334353 podStartE2EDuration="24.081576307s" totalImagesPullingTime="21.641241954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:21.081191305 +0000 UTC m=+25.759409089" watchObservedRunningTime="2026-07-23 09:17:21.081576307 +0000 UTC m=+25.759794092"
Jul 23 09:17:21.511875 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.511846    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:17:21.512427 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.512403    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-czvh8"
Jul 23 09:17:21.907245 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.907152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:21.907418 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:21.907280    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:21.907418 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:21.907343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:21.907541 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:21.907459    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:22.906992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:22.906747    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:22.907410 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:22.907087    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:23.906690 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:23.906654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:23.906820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:23.906692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:23.906820 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:23.906781    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:23.906936 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:23.906900    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:24.065397 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.065318    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="3986f36c748d65ae4955fa48c36fc9c78e888d6d0fcf7d56fd3d979e3a8fae32" exitCode=0
Jul 23 09:17:24.065397 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.065382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"3986f36c748d65ae4955fa48c36fc9c78e888d6d0fcf7d56fd3d979e3a8fae32"}
Jul 23 09:17:24.069241 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.069220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" event={"ID":"d72157fc-26b6-4c02-b1fc-8631f6a9d570","Type":"ContainerStarted","Data":"c8d09339176ebedfd87c26794d27cd1b0776d7097ff1e99bbed57a0fba0c6c13"}
Jul 23 09:17:24.069462 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.069450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:24.069510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.069488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:24.084526 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.084381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:24.230670 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.230627    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x" podStartSLOduration=7.6680857190000005 podStartE2EDuration="27.230615175s" totalImagesPullingTime="19.562529456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:24.227601542 +0000 UTC m=+28.905819346" watchObservedRunningTime="2026-07-23 09:17:24.230615175 +0000 UTC m=+28.908832958"
Jul 23 09:17:24.906964 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:24.906933    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:24.907175 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:24.907030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:25.071852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.071821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:25.085808 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.085787    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:25.535735 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.533561    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rq4z"]
Jul 23 09:17:25.535735 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.533720    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:25.535735 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:25.533855    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:25.542680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.542633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nmlww"]
Jul 23 09:17:25.542797 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.542738    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:25.542879 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:25.542846    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:25.543565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.543546    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk2bf"]
Jul 23 09:17:25.543640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:25.543629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:25.543733 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:25.543715    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:26.075456 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:26.075420    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="15becb708f5c66e2a379fb90472ca30199898227afd2fcfc064c3a34c8c858de" exitCode=0
Jul 23 09:17:26.075964 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:26.075508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"15becb708f5c66e2a379fb90472ca30199898227afd2fcfc064c3a34c8c858de"}
Jul 23 09:17:26.906318 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:26.906287    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:26.906487 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:26.906416    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:26.906487 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:26.906473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:26.906574 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:26.906560    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:27.909303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:27.909131    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:27.909717 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:27.909414    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:28.081960 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:28.081907    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="440c088ef3291f4d1a1e5578afe4606172a2ede9ab7a52e6c7f2e61418250877" exitCode=0
Jul 23 09:17:28.082111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:28.081972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"440c088ef3291f4d1a1e5578afe4606172a2ede9ab7a52e6c7f2e61418250877"}
Jul 23 09:17:28.906321 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:28.906290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:28.906321 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:28.906319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:28.906568 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:28.906399    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:28.906568 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:28.906530    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:29.455112 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:29.455081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:29.455574 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.455204    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:29.455574 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.455261    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs podName:4797eef2-0e43-426d-bbfb-70b818f55b01 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:01.455246227 +0000 UTC m=+66.133464001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs") pod "network-metrics-daemon-hk2bf" (UID: "4797eef2-0e43-426d-bbfb-70b818f55b01") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 09:17:29.555884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:29.555848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:29.556065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:29.555897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:29.556065 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556023    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:29.556065 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556037    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 09:17:29.556065 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556065    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 09:17:29.556223 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556078    2560 projected.go:196] Error preparing data for projected volume kube-api-access-gdvjq for pod openshift-network-diagnostics/network-check-target-nmlww: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:29.556223 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556089    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret podName:2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:01.556069549 +0000 UTC m=+66.234287313 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret") pod "global-pull-secret-syncer-9rq4z" (UID: "2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 09:17:29.556223 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.556120    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq podName:f97d932a-021e-445b-a5f6-73b954b3d32c nodeName:}" failed. No retries permitted until 2026-07-23 09:18:01.556105869 +0000 UTC m=+66.234323650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdvjq" (UniqueName: "kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq") pod "network-check-target-nmlww" (UID: "f97d932a-021e-445b-a5f6-73b954b3d32c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 09:17:29.910707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:29.910683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:29.910885 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:29.910799    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk2bf" podUID="4797eef2-0e43-426d-bbfb-70b818f55b01"
Jul 23 09:17:30.907056 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:30.907023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:30.907456 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:30.907142    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rq4z" podUID="2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5"
Jul 23 09:17:30.907456 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:30.907197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:30.907456 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:30.907288    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nmlww" podUID="f97d932a-021e-445b-a5f6-73b954b3d32c"
Jul 23 09:17:31.187281 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.187253    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-47.ec2.internal" event="NodeReady"
Jul 23 09:17:31.187447 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.187386    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 09:17:31.230487 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.229271    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675b9f7496-tdgxx"]
Jul 23 09:17:31.232059 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.232041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.234998 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.234980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 09:17:31.235507 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.235484    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 09:17:31.236033 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.236017    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5wx9\""
Jul 23 09:17:31.236307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.236292    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 09:17:31.258415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.258397    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cs9ms"]
Jul 23 09:17:31.259048 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.259034    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 09:17:31.260075 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.260057    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.262945 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.262906    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 09:17:31.263025 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.262991    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 09:17:31.263252 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.263232    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8qpt\""
Jul 23 09:17:31.272873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.272853    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jghtb"]
Jul 23 09:17:31.274709 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.274692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.278974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.278957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 09:17:31.279254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.279235    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 09:17:31.298843 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.298803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 09:17:31.298986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.298900    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bcmmh\""
Jul 23 09:17:31.299746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.299726    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675b9f7496-tdgxx"]
Jul 23 09:17:31.300549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.300480    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cs9ms"]
Jul 23 09:17:31.301103 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.301082    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghtb"]
Jul 23 09:17:31.368986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.368961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf121fd-4c05-4506-8496-e70b6f7e2bec-cert\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.369273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvz2\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3f90a5-4578-42d2-83fe-e104048cfaae-config-volume\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.369273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fxt9\" (UniqueName: \"kubernetes.io/projected/9b3f90a5-4578-42d2-83fe-e104048cfaae-kube-api-access-5fxt9\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.369273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369492 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369325    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369492 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbrg\" (UniqueName: \"kubernetes.io/projected/5cf121fd-4c05-4506-8496-e70b6f7e2bec-kube-api-access-lhbrg\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.369492 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3f90a5-4578-42d2-83fe-e104048cfaae-tmp-dir\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.369492 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.369492 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.369439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.470203 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470203 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf121fd-4c05-4506-8496-e70b6f7e2bec-cert\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvz2\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.470350    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:31.470381 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.470373    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:31.470642 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.470427    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:31.970408798 +0000 UTC m=+36.648626577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:31.470642 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3f90a5-4578-42d2-83fe-e104048cfaae-config-volume\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.470642 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fxt9\" (UniqueName: \"kubernetes.io/projected/9b3f90a5-4578-42d2-83fe-e104048cfaae-kube-api-access-5fxt9\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.470642 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbrg\" (UniqueName: \"kubernetes.io/projected/5cf121fd-4c05-4506-8496-e70b6f7e2bec-kube-api-access-lhbrg\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.470824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470717    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3f90a5-4578-42d2-83fe-e104048cfaae-tmp-dir\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.470824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.470824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.470768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.471089 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.470867    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:31.471089 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.470933    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:17:31.970895734 +0000 UTC m=+36.649113499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:31.471089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.471038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.471222 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.471153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3f90a5-4578-42d2-83fe-e104048cfaae-config-volume\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.471222 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.471197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3f90a5-4578-42d2-83fe-e104048cfaae-tmp-dir\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.471298 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.471276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.471364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.471339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.475180 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.475159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.475180 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.475172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.475302 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.475269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5cf121fd-4c05-4506-8496-e70b6f7e2bec-cert\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.484844 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.484799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.485086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.485042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvz2\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.486740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.486719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fxt9\" (UniqueName: \"kubernetes.io/projected/9b3f90a5-4578-42d2-83fe-e104048cfaae-kube-api-access-5fxt9\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.487089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.487069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbrg\" (UniqueName: \"kubernetes.io/projected/5cf121fd-4c05-4506-8496-e70b6f7e2bec-kube-api-access-lhbrg\") pod \"ingress-canary-jghtb\" (UID: \"5cf121fd-4c05-4506-8496-e70b6f7e2bec\") " pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.583858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.583832    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jghtb"
Jul 23 09:17:31.906632 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.906553    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:17:31.909550 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.909503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 09:17:31.910169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.909608    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tghcj\""
Jul 23 09:17:31.975483 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.975449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:31.975668 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:31.975528    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:31.975668 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.975599    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:31.975668 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.975637    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:31.975668 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.975647    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:31.975819 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.975685    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:17:32.975665445 +0000 UTC m=+37.653883225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:31.975819 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:31.975702    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:32.975694587 +0000 UTC m=+37.653912348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:32.177933 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.177887    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jghtb"]
Jul 23 09:17:32.181014 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:17:32.180991    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf121fd_4c05_4506_8496_e70b6f7e2bec.slice/crio-cb2554a892d271d69746e3302df7d0fb99c338a3ac81589cc0ba12154fa832d1 WatchSource:0}: Error finding container cb2554a892d271d69746e3302df7d0fb99c338a3ac81589cc0ba12154fa832d1: Status 404 returned error can't find the container with id cb2554a892d271d69746e3302df7d0fb99c338a3ac81589cc0ba12154fa832d1
Jul 23 09:17:32.906728 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.906696    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:17:32.906898 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.906700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:17:32.909566 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.909536    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 09:17:32.909566 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.909539    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 09:17:32.910516 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.910494    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2sdc\""
Jul 23 09:17:32.910604 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.910538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 09:17:32.982744 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.982705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:32.982891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:32.982784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:32.982891 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:32.982855    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:32.982891 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:32.982885    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:32.983039 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:32.982898    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:32.983039 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:32.982940    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:17:34.98290806 +0000 UTC m=+39.661125827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:32.983039 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:32.982959    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:34.982949982 +0000 UTC m=+39.661167745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:33.095723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:33.095688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghtb" event={"ID":"5cf121fd-4c05-4506-8496-e70b6f7e2bec","Type":"ContainerStarted","Data":"cb2554a892d271d69746e3302df7d0fb99c338a3ac81589cc0ba12154fa832d1"}
Jul 23 09:17:33.098904 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:33.098875    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="2510d937c41315bd22569fd56bad5b784a556bb95c958ec56bf6c0d277404455" exitCode=0
Jul 23 09:17:33.099041 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:33.098910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"2510d937c41315bd22569fd56bad5b784a556bb95c958ec56bf6c0d277404455"}
Jul 23 09:17:34.105071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:34.105037    2560 generic.go:356] "Generic (PLEG): container finished" podID="460e979e-3746-4b38-8768-779560aba737" containerID="9bdc31411bb57dbce39085a36443f375ce041fa2c50905d85f5e436ecc2d9cca" exitCode=0
Jul 23 09:17:34.105547 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:34.105080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerDied","Data":"9bdc31411bb57dbce39085a36443f375ce041fa2c50905d85f5e436ecc2d9cca"}
Jul 23 09:17:34.998009 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:34.997813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:34.998156 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:34.998050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:34.998156 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:34.997984    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:34.998156 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:34.998152    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:34.998254 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:34.998164    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:34.998254 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:34.998152    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.998135625 +0000 UTC m=+43.676353386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:34.998254 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:34.998211    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.998197737 +0000 UTC m=+43.676415501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:35.110148 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:35.110120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" event={"ID":"460e979e-3746-4b38-8768-779560aba737","Type":"ContainerStarted","Data":"7276fa4780990136685273d58cc9cb71056501038259064ecc3c2de210133ccc"}
Jul 23 09:17:35.111460 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:35.111437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jghtb" event={"ID":"5cf121fd-4c05-4506-8496-e70b6f7e2bec","Type":"ContainerStarted","Data":"15bceff8c481b1060a7ed6c340a7566819d06674e9080796ae7fdc336d23b466"}
Jul 23 09:17:35.137467 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:35.137425    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rl4q7" podStartSLOduration=9.646384112 podStartE2EDuration="39.137411972s" totalImagesPullingTime="29.49102786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:35.135992096 +0000 UTC m=+39.814209901" watchObservedRunningTime="2026-07-23 09:17:35.137411972 +0000 UTC m=+39.815629755"
Jul 23 09:17:35.153472 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:35.153436    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jghtb" podStartSLOduration=1.8677961870000002 podStartE2EDuration="4.153424349s" totalImagesPullingTime="2.285628162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:17:35.153048467 +0000 UTC m=+39.831266264" watchObservedRunningTime="2026-07-23 09:17:35.153424349 +0000 UTC m=+39.831642129"
Jul 23 09:17:35.986739 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:35.986706    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"]
Jul 23 09:17:36.026001 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.025977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"]
Jul 23 09:17:36.026123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.026058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.028958 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.028939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 23 09:17:36.028958 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.028954    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k9q99\""
Jul 23 09:17:36.029856 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.029833    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 23 09:17:36.205987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.205967    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.206276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.206009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c22ced1f-2f2d-47e5-98e1-e70166d53f22-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.306516 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.306462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.306516 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.306506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c22ced1f-2f2d-47e5-98e1-e70166d53f22-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.306653 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:36.306603    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:36.306714 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:36.306670    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:36.80664833 +0000 UTC m=+41.484866092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:36.307199 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.307175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c22ced1f-2f2d-47e5-98e1-e70166d53f22-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.810096 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:36.810072    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:36.810222 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:36.810197    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:36.810274 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:36.810252    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:37.810238997 +0000 UTC m=+42.488456763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:37.144946 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.144863    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ft8xt"]
Jul 23 09:17:37.169928 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.169892    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ft8xt"]
Jul 23 09:17:37.170060 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.170035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.173024 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.173006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 09:17:37.173111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.173015    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 09:17:37.173111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.173042    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 09:17:37.174355 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.174337    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 09:17:37.174460 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.174388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sjj26\""
Jul 23 09:17:37.313921 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.313900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a6b63b1-e2b7-4cab-ba7f-94c406853555-crio-socket\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.314187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.313943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5qg\" (UniqueName: \"kubernetes.io/projected/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-api-access-ms5qg\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.314187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.314044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a6b63b1-e2b7-4cab-ba7f-94c406853555-data-volume\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.314187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.314077    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.314187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.314100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415395 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a6b63b1-e2b7-4cab-ba7f-94c406853555-crio-socket\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415544 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.415497    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 09:17:37.415576 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.415558    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:37.915540768 +0000 UTC m=+42.593758549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : configmap "kube-rbac-proxy" not found
Jul 23 09:17:37.415612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5qg\" (UniqueName: \"kubernetes.io/projected/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-api-access-ms5qg\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415664 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.415647    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:37.415700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415588    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a6b63b1-e2b7-4cab-ba7f-94c406853555-crio-socket\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415738 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a6b63b1-e2b7-4cab-ba7f-94c406853555-data-volume\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.415738 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.415709    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:37.915692251 +0000 UTC m=+42.593910028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:37.415978 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.415963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a6b63b1-e2b7-4cab-ba7f-94c406853555-data-volume\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.426560 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.426535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5qg\" (UniqueName: \"kubernetes.io/projected/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-api-access-ms5qg\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.818205 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.818137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:37.818309 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.818274    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:37.821099 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.819181    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:39.819157405 +0000 UTC m=+44.497375170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:37.918837 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.918814    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.918954 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:37.918845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:37.918954 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.918952    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:37.919056 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.918960    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 09:17:37.919056 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.919001    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.918990411 +0000 UTC m=+43.597208171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:37.919056 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:37.919018    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:38.919007227 +0000 UTC m=+43.597224989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : configmap "kube-rbac-proxy" not found
Jul 23 09:17:38.926554 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:38.926519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:38.926554 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:38.926557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:38.927109 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:38.926650    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 09:17:38.927109 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:38.926681    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:38.927109 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:38.926732    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:40.926710023 +0000 UTC m=+45.604927801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : configmap "kube-rbac-proxy" not found
Jul 23 09:17:38.927109 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:38.926751    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:40.926741973 +0000 UTC m=+45.604959740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:39.027121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:39.027093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:39.027232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:39.027170    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:39.027232 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.027208    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:39.027352 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.027249    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:17:47.02723033 +0000 UTC m=+51.705448097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:39.027352 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.027276    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:39.027352 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.027289    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:39.027352 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.027342    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:47.027327715 +0000 UTC m=+51.705545479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:39.429375 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:39.429353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7lc8_e6e5add7-e2a7-47a6-a2e9-8685a6af4dff/dns-node-resolver/0.log"
Jul 23 09:17:39.832126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:39.832046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:39.832263 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.832197    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:39.832298 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:39.832264    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:43.832247975 +0000 UTC m=+48.510465755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:40.429615 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:40.429590    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xzzhs_f744bf95-832b-4e38-8967-3f794694cdbd/node-ca/0.log"
Jul 23 09:17:40.940499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:40.940474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:40.940663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:40.940509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:40.940663 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:40.940597    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 09:17:40.940663 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:40.940607    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:40.940663 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:40.940656    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:44.940643525 +0000 UTC m=+49.618861285 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:40.940870 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:40.940701    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:44.940685909 +0000 UTC m=+49.618903670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : configmap "kube-rbac-proxy" not found
Jul 23 09:17:41.032040 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:41.032018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jghtb_5cf121fd-4c05-4506-8496-e70b6f7e2bec/serve-healthcheck-canary/0.log"
Jul 23 09:17:43.862071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:43.862039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:43.862515 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:43.862156    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:43.862515 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:43.862207    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:51.86219345 +0000 UTC m=+56.540411212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:44.971307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:44.971275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:44.971307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:44.971312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:44.971703 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:44.971455    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:44.971703 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:44.971505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls podName:0a6b63b1-e2b7-4cab-ba7f-94c406853555 nodeName:}" failed. No retries permitted until 2026-07-23 09:17:52.971492545 +0000 UTC m=+57.649710306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ft8xt" (UID: "0a6b63b1-e2b7-4cab-ba7f-94c406853555") : secret "insights-runtime-extractor-tls" not found
Jul 23 09:17:44.971781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:44.971757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a6b63b1-e2b7-4cab-ba7f-94c406853555-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:47.085109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:47.085074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:47.085139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:47.085206    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:47.085217    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:47.085222    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b9f7496-tdgxx: secret "image-registry-tls" not found
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:47.085263    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls podName:9b3f90a5-4578-42d2-83fe-e104048cfaae nodeName:}" failed. No retries permitted until 2026-07-23 09:18:03.085250162 +0000 UTC m=+67.763467923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls") pod "dns-default-cs9ms" (UID: "9b3f90a5-4578-42d2-83fe-e104048cfaae") : secret "dns-default-metrics-tls" not found
Jul 23 09:17:47.085471 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:47.085275    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls podName:196162a4-ed18-4a66-8ea4-20cfd8a86239 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:03.085269429 +0000 UTC m=+67.763487190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls") pod "image-registry-675b9f7496-tdgxx" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239") : secret "image-registry-tls" not found
Jul 23 09:17:51.920839 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:51.920805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:17:51.921302 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:51.920969    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 23 09:17:51.921302 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:17:51.921029    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert podName:c22ced1f-2f2d-47e5-98e1-e70166d53f22 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:07.921011071 +0000 UTC m=+72.599228849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-nbb7b" (UID: "c22ced1f-2f2d-47e5-98e1-e70166d53f22") : secret "networking-console-plugin-cert" not found
Jul 23 09:17:53.028452 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:53.028418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:53.030746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:53.030723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a6b63b1-e2b7-4cab-ba7f-94c406853555-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ft8xt\" (UID: \"0a6b63b1-e2b7-4cab-ba7f-94c406853555\") " pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:53.079629 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:53.079594    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ft8xt"
Jul 23 09:17:53.205282 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:53.205252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ft8xt"]
Jul 23 09:17:53.208396 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:17:53.208369    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6b63b1_e2b7_4cab_ba7f_94c406853555.slice/crio-48a2660a6a98ac84698a80e9a2db51df3f5d54e541e23c4bd2041a7f6c374406 WatchSource:0}: Error finding container 48a2660a6a98ac84698a80e9a2db51df3f5d54e541e23c4bd2041a7f6c374406: Status 404 returned error can't find the container with id 48a2660a6a98ac84698a80e9a2db51df3f5d54e541e23c4bd2041a7f6c374406
Jul 23 09:17:54.156071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:54.156031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft8xt" event={"ID":"0a6b63b1-e2b7-4cab-ba7f-94c406853555","Type":"ContainerStarted","Data":"48a2660a6a98ac84698a80e9a2db51df3f5d54e541e23c4bd2041a7f6c374406"}
Jul 23 09:17:57.090679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:57.090633    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mnc5x"
Jul 23 09:17:58.165946 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:17:58.165862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft8xt" event={"ID":"0a6b63b1-e2b7-4cab-ba7f-94c406853555","Type":"ContainerStarted","Data":"25f7be06ac16eff4ca0be576489f156de8fe399aea2811c9ffaf53daa43f7762"}
Jul 23 09:18:01.492186 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.492161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:18:01.495122 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.495101    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 09:18:01.504465 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.504442    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4797eef2-0e43-426d-bbfb-70b818f55b01-metrics-certs\") pod \"network-metrics-daemon-hk2bf\" (UID: \"4797eef2-0e43-426d-bbfb-70b818f55b01\") " pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:18:01.593296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.593274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:18:01.593382 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.593302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:18:01.595853 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.595835    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 09:18:01.596071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.596058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 09:18:01.605808 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.605786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5-original-pull-secret\") pod \"global-pull-secret-syncer-9rq4z\" (UID: \"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5\") " pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:18:01.606367 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.606353    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 09:18:01.616335 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.616314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdvjq\" (UniqueName: \"kubernetes.io/projected/f97d932a-021e-445b-a5f6-73b954b3d32c-kube-api-access-gdvjq\") pod \"network-check-target-nmlww\" (UID: \"f97d932a-021e-445b-a5f6-73b954b3d32c\") " pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:18:01.618535 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.618520    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tghcj\""
Jul 23 09:18:01.626659 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.626646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk2bf"
Jul 23 09:18:01.718310 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.718283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rq4z"
Jul 23 09:18:01.727135 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.727113    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s2sdc\""
Jul 23 09:18:01.734564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.734542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:18:01.747596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.747565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk2bf"]
Jul 23 09:18:01.750880 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:01.750856    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4797eef2_0e43_426d_bbfb_70b818f55b01.slice/crio-124cf0184a7363a764b70f1a63363be22d4bc9644e4ba72cef60d78c45bf09b8 WatchSource:0}: Error finding container 124cf0184a7363a764b70f1a63363be22d4bc9644e4ba72cef60d78c45bf09b8: Status 404 returned error can't find the container with id 124cf0184a7363a764b70f1a63363be22d4bc9644e4ba72cef60d78c45bf09b8
Jul 23 09:18:01.850970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.850943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rq4z"]
Jul 23 09:18:01.853832 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:01.853805    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4b6b60_ef78_4bd4_b917_c3271a1fdbd5.slice/crio-22d5cab7283b4fd1e25b5a0364ddba3b16e1065a39b97e2c6e71dbe0a65c8307 WatchSource:0}: Error finding container 22d5cab7283b4fd1e25b5a0364ddba3b16e1065a39b97e2c6e71dbe0a65c8307: Status 404 returned error can't find the container with id 22d5cab7283b4fd1e25b5a0364ddba3b16e1065a39b97e2c6e71dbe0a65c8307
Jul 23 09:18:01.864271 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:01.864250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nmlww"]
Jul 23 09:18:01.866791 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:01.866764    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97d932a_021e_445b_a5f6_73b954b3d32c.slice/crio-6291e757f12229627e2f63496439b5a4e931ecc020399e60656c7fbfc0ed50b0 WatchSource:0}: Error finding container 6291e757f12229627e2f63496439b5a4e931ecc020399e60656c7fbfc0ed50b0: Status 404 returned error can't find the container with id 6291e757f12229627e2f63496439b5a4e931ecc020399e60656c7fbfc0ed50b0
Jul 23 09:18:02.177627 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.177540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk2bf" event={"ID":"4797eef2-0e43-426d-bbfb-70b818f55b01","Type":"ContainerStarted","Data":"124cf0184a7363a764b70f1a63363be22d4bc9644e4ba72cef60d78c45bf09b8"}
Jul 23 09:18:02.179647 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.179617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft8xt" event={"ID":"0a6b63b1-e2b7-4cab-ba7f-94c406853555","Type":"ContainerStarted","Data":"86f5fd3feb0a6884b359e0eb2bc2b1ef413386ec3437cad74eb3014c441dca24"}
Jul 23 09:18:02.180784 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.180759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rq4z" event={"ID":"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5","Type":"ContainerStarted","Data":"22d5cab7283b4fd1e25b5a0364ddba3b16e1065a39b97e2c6e71dbe0a65c8307"}
Jul 23 09:18:02.181876 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.181841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nmlww" event={"ID":"f97d932a-021e-445b-a5f6-73b954b3d32c","Type":"ContainerStarted","Data":"6291e757f12229627e2f63496439b5a4e931ecc020399e60656c7fbfc0ed50b0"}
Jul 23 09:18:02.760726 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.760688    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"]
Jul 23 09:18:02.793409 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.793362    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"]
Jul 23 09:18:02.793559 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.793510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:02.796462 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.796306    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b9f7496-tdgxx"]
Jul 23 09:18:02.796595 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:02.796510    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-675b9f7496-tdgxx" podUID="196162a4-ed18-4a66-8ea4-20cfd8a86239"
Jul 23 09:18:02.799360 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.799228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 09:18:02.812984 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.812702    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 23 09:18:02.813088 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.812985    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 09:18:02.813462 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.813261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pn7q9\""
Jul 23 09:18:02.813564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.813504    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 09:18:02.906854 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.906813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f2ca65d-379f-4d15-b5d8-012d644eff5b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:02.907032 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.906871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64njr\" (UniqueName: \"kubernetes.io/projected/1f2ca65d-379f-4d15-b5d8-012d644eff5b-kube-api-access-64njr\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:02.971989 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.971940    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c6c89897c-2zg9x"]
Jul 23 09:18:02.983799 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:02.983774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.007876 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.007848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64njr\" (UniqueName: \"kubernetes.io/projected/1f2ca65d-379f-4d15-b5d8-012d644eff5b-kube-api-access-64njr\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:03.008017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.007981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f2ca65d-379f-4d15-b5d8-012d644eff5b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:03.011272 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.011204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f2ca65d-379f-4d15-b5d8-012d644eff5b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:03.045751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.045654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6c89897c-2zg9x"]
Jul 23 09:18:03.085218 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.085188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64njr\" (UniqueName: \"kubernetes.io/projected/1f2ca65d-379f-4d15-b5d8-012d644eff5b-kube-api-access-64njr\") pod \"managed-serviceaccount-addon-agent-bdf546f9d-h6zxs\" (UID: \"1f2ca65d-379f-4d15-b5d8-012d644eff5b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:03.108992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.108963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-tls\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109137 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-trusted-ca\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109137 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-ca-trust-extracted\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109137 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-bound-sa-token\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109137 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:18:03.109366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-installation-pull-secrets\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbghn\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-kube-api-access-fbghn\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:18:03.109366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-certificates\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.109564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.109391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-image-registry-private-configuration\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.112778 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.112578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b3f90a5-4578-42d2-83fe-e104048cfaae-metrics-tls\") pod \"dns-default-cs9ms\" (UID: \"9b3f90a5-4578-42d2-83fe-e104048cfaae\") " pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:18:03.113080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.113030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"image-registry-675b9f7496-tdgxx\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") " pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:18:03.115017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.114661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"
Jul 23 09:18:03.184743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.184655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:18:03.209803 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.209779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-tls\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.209958 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.209828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-trusted-ca\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.209958 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.209858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-ca-trust-extracted\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.209958 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.209885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-bound-sa-token\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.210128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.209972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-installation-pull-secrets\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.210128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.210036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbghn\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-kube-api-access-fbghn\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.210128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.210073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-certificates\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.210128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.210118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-image-registry-private-configuration\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.210627 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.210602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-ca-trust-extracted\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.211351 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.211308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-trusted-ca\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.211686 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.211644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-certificates\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.211903 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.211826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:18:03.213033 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.213009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-registry-tls\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.222047 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.216027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-installation-pull-secrets\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.222047 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.216406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-image-registry-private-configuration\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.225788 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.225742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-bound-sa-token\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.228526 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.228505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbghn\" (UniqueName: \"kubernetes.io/projected/85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9-kube-api-access-fbghn\") pod \"image-registry-7c6c89897c-2zg9x\" (UID: \"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9\") " pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.297557 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.297455    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5wx9\""
Jul 23 09:18:03.305960 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.305927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:03.310781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.310891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310799    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.310891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.310891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310878    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.311696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.311696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310949    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.311696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.310976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwvz2\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.311696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.311011    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates\") pod \"196162a4-ed18-4a66-8ea4-20cfd8a86239\" (UID: \"196162a4-ed18-4a66-8ea4-20cfd8a86239\") "
Jul 23 09:18:03.311696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.311504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:18:03.312102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.312064    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:18:03.312273 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.312239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:18:03.314048 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.314003    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:18:03.314355 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.314312    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:18:03.314751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.314727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:18:03.315728 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.315701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:18:03.316675 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.316646    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2" (OuterVolumeSpecName: "kube-api-access-hwvz2") pod "196162a4-ed18-4a66-8ea4-20cfd8a86239" (UID: "196162a4-ed18-4a66-8ea4-20cfd8a86239"). InnerVolumeSpecName "kube-api-access-hwvz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:18:03.376279 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.376250    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p8qpt\""
Jul 23 09:18:03.381799 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.381595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413607    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-trusted-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413634    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/196162a4-ed18-4a66-8ea4-20cfd8a86239-ca-trust-extracted\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413649    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413664    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-installation-pull-secrets\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413679    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvz2\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-kube-api-access-hwvz2\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413692    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/196162a4-ed18-4a66-8ea4-20cfd8a86239-registry-certificates\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413705    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/196162a4-ed18-4a66-8ea4-20cfd8a86239-bound-sa-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.413746 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.413719    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/196162a4-ed18-4a66-8ea4-20cfd8a86239-image-registry-private-configuration\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:18:03.520134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.519945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs"]
Jul 23 09:18:03.524129 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:03.524091    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f2ca65d_379f_4d15_b5d8_012d644eff5b.slice/crio-a09d2a6d9195bc42d611efbef7fde3cb06883d777dbcb17e2a73b81e49504d4a WatchSource:0}: Error finding container a09d2a6d9195bc42d611efbef7fde3cb06883d777dbcb17e2a73b81e49504d4a: Status 404 returned error can't find the container with id a09d2a6d9195bc42d611efbef7fde3cb06883d777dbcb17e2a73b81e49504d4a
Jul 23 09:18:03.565126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.565018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c6c89897c-2zg9x"]
Jul 23 09:18:03.570528 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:03.570496    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e5b1e4_1fef_45d9_b7d6_b1a35de9a7e9.slice/crio-7f9b445211bc7260b391431aa549c75fdae89b67ea330e34f2c8b4f415e8b758 WatchSource:0}: Error finding container 7f9b445211bc7260b391431aa549c75fdae89b67ea330e34f2c8b4f415e8b758: Status 404 returned error can't find the container with id 7f9b445211bc7260b391431aa549c75fdae89b67ea330e34f2c8b4f415e8b758
Jul 23 09:18:03.600127 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:03.600030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cs9ms"]
Jul 23 09:18:03.607613 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:03.607587    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3f90a5_4578_42d2_83fe_e104048cfaae.slice/crio-4a50b49e94e00c79cecf046877375b30b0b25f75b825e4864a3c1f6c9af549a2 WatchSource:0}: Error finding container 4a50b49e94e00c79cecf046877375b30b0b25f75b825e4864a3c1f6c9af549a2: Status 404 returned error can't find the container with id 4a50b49e94e00c79cecf046877375b30b0b25f75b825e4864a3c1f6c9af549a2
Jul 23 09:18:04.191533 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.191444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk2bf" event={"ID":"4797eef2-0e43-426d-bbfb-70b818f55b01","Type":"ContainerStarted","Data":"d63cda4ba5bf0edcb530bab20f32d4c3f68d04225503225310960c47980eb76f"}
Jul 23 09:18:04.191533 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.191513    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk2bf" event={"ID":"4797eef2-0e43-426d-bbfb-70b818f55b01","Type":"ContainerStarted","Data":"daaad57db113bb8bbf68689a02cc9e7f93d75cd1b792a3a2c050fa66cb14368e"}
Jul 23 09:18:04.193887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.193858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs9ms" event={"ID":"9b3f90a5-4578-42d2-83fe-e104048cfaae","Type":"ContainerStarted","Data":"4a50b49e94e00c79cecf046877375b30b0b25f75b825e4864a3c1f6c9af549a2"}
Jul 23 09:18:04.196082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.196054    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x" event={"ID":"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9","Type":"ContainerStarted","Data":"a4040756a84125c646014bc85bdc7cb4478329523164d370d3f204e306f4e65e"}
Jul 23 09:18:04.196217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.196104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x" event={"ID":"85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9","Type":"ContainerStarted","Data":"7f9b445211bc7260b391431aa549c75fdae89b67ea330e34f2c8b4f415e8b758"}
Jul 23 09:18:04.196217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.196131    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:04.198231 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.197646    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b9f7496-tdgxx"
Jul 23 09:18:04.198231 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.198108    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs" event={"ID":"1f2ca65d-379f-4d15-b5d8-012d644eff5b","Type":"ContainerStarted","Data":"a09d2a6d9195bc42d611efbef7fde3cb06883d777dbcb17e2a73b81e49504d4a"}
Jul 23 09:18:04.223696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.223307    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hk2bf" podStartSLOduration=66.662837904 podStartE2EDuration="1m8.223288433s" totalImagesPullingTime="1.560450529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:04.220313433 +0000 UTC m=+68.898531218" watchObservedRunningTime="2026-07-23 09:18:04.223288433 +0000 UTC m=+68.901506218"
Jul 23 09:18:04.297971 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.297905    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x" podStartSLOduration=2.297880873 podStartE2EDuration="2.297880873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:04.260761869 +0000 UTC m=+68.938979652" watchObservedRunningTime="2026-07-23 09:18:04.297880873 +0000 UTC m=+68.976098657"
Jul 23 09:18:04.315289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.315238    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b9f7496-tdgxx"]
Jul 23 09:18:04.335741 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:04.335695    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675b9f7496-tdgxx"]
Jul 23 09:18:05.910776 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:05.910736    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196162a4-ed18-4a66-8ea4-20cfd8a86239" path="/var/lib/kubelet/pods/196162a4-ed18-4a66-8ea4-20cfd8a86239/volumes"
Jul 23 09:18:06.614528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.614498    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"]
Jul 23 09:18:06.616529 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.616508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:06.620055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.620036    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gv46b\""
Jul 23 09:18:06.636104 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.636074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 23 09:18:06.636314 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.636297    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"]
Jul 23 09:18:06.741019 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.740979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mxm6v\" (UID: \"699488dc-2e27-4a52-bb09-36e7718ee085\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:06.842223 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:06.842185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mxm6v\" (UID: \"699488dc-2e27-4a52-bb09-36e7718ee085\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:06.842368 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:06.842313    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 09:18:06.842420 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:06.842393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates podName:699488dc-2e27-4a52-bb09-36e7718ee085 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:07.342372436 +0000 UTC m=+72.020590201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-mxm6v" (UID: "699488dc-2e27-4a52-bb09-36e7718ee085") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 23 09:18:07.345888 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:07.345828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mxm6v\" (UID: \"699488dc-2e27-4a52-bb09-36e7718ee085\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:07.348866 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:07.348836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699488dc-2e27-4a52-bb09-36e7718ee085-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-mxm6v\" (UID: \"699488dc-2e27-4a52-bb09-36e7718ee085\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:07.526465 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:07.526432    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:07.951795 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:07.951764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:18:07.954320 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:07.954293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c22ced1f-2f2d-47e5-98e1-e70166d53f22-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-nbb7b\" (UID: \"c22ced1f-2f2d-47e5-98e1-e70166d53f22\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:18:08.137803 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:08.137773    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k9q99\""
Jul 23 09:18:08.145556 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:08.145531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"
Jul 23 09:18:08.969955 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:08.969310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b"]
Jul 23 09:18:08.975300 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:08.975264    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22ced1f_2f2d_47e5_98e1_e70166d53f22.slice/crio-03cf8c11e61f2657ea6aaf0aec2effc31945f9e69a94ad2356d0497fb1f73a42 WatchSource:0}: Error finding container 03cf8c11e61f2657ea6aaf0aec2effc31945f9e69a94ad2356d0497fb1f73a42: Status 404 returned error can't find the container with id 03cf8c11e61f2657ea6aaf0aec2effc31945f9e69a94ad2356d0497fb1f73a42
Jul 23 09:18:08.990131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:08.990026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"]
Jul 23 09:18:08.995726 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:08.995697    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699488dc_2e27_4a52_bb09_36e7718ee085.slice/crio-29a00fc065d3ae993e55c68761095cb3fd917547463899e2327b2f96373bcb1c WatchSource:0}: Error finding container 29a00fc065d3ae993e55c68761095cb3fd917547463899e2327b2f96373bcb1c: Status 404 returned error can't find the container with id 29a00fc065d3ae993e55c68761095cb3fd917547463899e2327b2f96373bcb1c
Jul 23 09:18:09.217743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.217706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v" event={"ID":"699488dc-2e27-4a52-bb09-36e7718ee085","Type":"ContainerStarted","Data":"29a00fc065d3ae993e55c68761095cb3fd917547463899e2327b2f96373bcb1c"}
Jul 23 09:18:09.220016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.219983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ft8xt" event={"ID":"0a6b63b1-e2b7-4cab-ba7f-94c406853555","Type":"ContainerStarted","Data":"aa254403f7a4dfd91d294fc8e9d722532aa25f85d1dff50c2177873d3d4d9082"}
Jul 23 09:18:09.221480 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.221446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rq4z" event={"ID":"2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5","Type":"ContainerStarted","Data":"31832bf1c93438e456eab5c73f27db22ccce3da7e1dded2d1442ef80c076fb1e"}
Jul 23 09:18:09.222736 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.222709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b" event={"ID":"c22ced1f-2f2d-47e5-98e1-e70166d53f22","Type":"ContainerStarted","Data":"03cf8c11e61f2657ea6aaf0aec2effc31945f9e69a94ad2356d0497fb1f73a42"}
Jul 23 09:18:09.224424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.224284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nmlww" event={"ID":"f97d932a-021e-445b-a5f6-73b954b3d32c","Type":"ContainerStarted","Data":"39ff6d075e11b0da2b4a1db18917d8d7e77694d78fe9306cc6779489e1d68fa6"}
Jul 23 09:18:09.224424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.224377    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:18:09.226326 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.226306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs9ms" event={"ID":"9b3f90a5-4578-42d2-83fe-e104048cfaae","Type":"ContainerStarted","Data":"92fbd53622191375dbea660b507f1f40a1cce779a6d465537ae6a922220eb9d6"}
Jul 23 09:18:09.226416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.226340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cs9ms" event={"ID":"9b3f90a5-4578-42d2-83fe-e104048cfaae","Type":"ContainerStarted","Data":"8efc3c51b54360847675999de58c1e3ac769e343fbb6f4c603a34ab184040347"}
Jul 23 09:18:09.226465 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.226440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:18:09.243344 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.243303    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ft8xt" podStartSLOduration=16.747185713 podStartE2EDuration="32.243291323s" totalImagesPullingTime="15.49610561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:17:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:09.242675715 +0000 UTC m=+73.920893496" watchObservedRunningTime="2026-07-23 09:18:09.243291323 +0000 UTC m=+73.921509128"
Jul 23 09:18:09.294067 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.294012    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cs9ms" podStartSLOduration=33.055906567 podStartE2EDuration="38.2939971s" totalImagesPullingTime="5.238090533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:09.267183454 +0000 UTC m=+73.945401241" watchObservedRunningTime="2026-07-23 09:18:09.2939971 +0000 UTC m=+73.972214882"
Jul 23 09:18:09.294227 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.294125    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9rq4z" podStartSLOduration=65.347134723 podStartE2EDuration="1m12.294120224s" totalImagesPullingTime="6.946985501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 09:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:09.292633622 +0000 UTC m=+73.970851407" watchObservedRunningTime="2026-07-23 09:18:09.294120224 +0000 UTC m=+73.972337985"
Jul 23 09:18:09.314548 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:09.314456    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nmlww" podStartSLOduration=66.381403011 podStartE2EDuration="1m13.314438128s" totalImagesPullingTime="6.933035117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:16:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:09.314096408 +0000 UTC m=+73.992314215" watchObservedRunningTime="2026-07-23 09:18:09.314438128 +0000 UTC m=+73.992655901"
Jul 23 09:18:11.234596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.234518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b" event={"ID":"c22ced1f-2f2d-47e5-98e1-e70166d53f22","Type":"ContainerStarted","Data":"40bf164951364b2a3544b68b84b04ea20fc175645701ac39098d372d5750cb2e"}
Jul 23 09:18:11.236089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.236059    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v" event={"ID":"699488dc-2e27-4a52-bb09-36e7718ee085","Type":"ContainerStarted","Data":"7ba443dbfd2b737d1d277fec67afaf6351c31666305d87dca3bbf8a69dfc590f"}
Jul 23 09:18:11.236209 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.236180    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:11.237639 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.237612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs" event={"ID":"1f2ca65d-379f-4d15-b5d8-012d644eff5b","Type":"ContainerStarted","Data":"9ba33cb748db6ff3bbd71fc003093c278c73d5e004b48947eebc4c0456ea6203"}
Jul 23 09:18:11.240843 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.240825    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v"
Jul 23 09:18:11.260093 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.260055    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-nbb7b" podStartSLOduration=34.67953393 podStartE2EDuration="36.260041493s" totalImagesPullingTime="1.580507563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:17:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:11.259539996 +0000 UTC m=+75.937757807" watchObservedRunningTime="2026-07-23 09:18:11.260041493 +0000 UTC m=+75.938259341"
Jul 23 09:18:11.301243 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.301195    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-mxm6v" podStartSLOduration=3.368425871 podStartE2EDuration="5.301177324s" totalImagesPullingTime="1.932751453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:11.300566328 +0000 UTC m=+75.978784113" watchObservedRunningTime="2026-07-23 09:18:11.301177324 +0000 UTC m=+75.979395108"
Jul 23 09:18:11.302594 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.302558    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-bdf546f9d-h6zxs" podStartSLOduration=2.3071370030000002 podStartE2EDuration="9.302548329s" totalImagesPullingTime="6.995411326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:11.282362113 +0000 UTC m=+75.960579896" watchObservedRunningTime="2026-07-23 09:18:11.302548329 +0000 UTC m=+75.980766113"
Jul 23 09:18:11.618276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.618211    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"]
Jul 23 09:18:11.621306 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.621288    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.624467 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.624432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 23 09:18:11.624578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.624536    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 23 09:18:11.624640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.624540    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 09:18:11.625999 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.625978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vzbwh\""
Jul 23 09:18:11.626110 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.626050    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 09:18:11.626191 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.626176    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 09:18:11.634567 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.634544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"]
Jul 23 09:18:11.682856 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.682833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.682972 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.682873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgmj\" (UniqueName: \"kubernetes.io/projected/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-kube-api-access-2tgmj\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.682972 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.682894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.682972 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.682925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.784192 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.784167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.784257 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.784224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgmj\" (UniqueName: \"kubernetes.io/projected/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-kube-api-access-2tgmj\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.784257 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.784249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.784355 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.784266    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.784406 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:11.784354    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 23 09:18:11.784455 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:11.784406    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls podName:f1101cc3-0d14-489a-af1e-e38f6e3cbe67 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:12.284392164 +0000 UTC m=+76.962609926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-8lhnr" (UID: "f1101cc3-0d14-489a-af1e-e38f6e3cbe67") : secret "prometheus-operator-tls" not found
Jul 23 09:18:11.784759 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.784739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.786671 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.786650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:11.797760 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:11.797734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgmj\" (UniqueName: \"kubernetes.io/projected/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-kube-api-access-2tgmj\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:12.287564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:12.287535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:12.289652 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:12.289632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f1101cc3-0d14-489a-af1e-e38f6e3cbe67-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8lhnr\" (UID: \"f1101cc3-0d14-489a-af1e-e38f6e3cbe67\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:12.530637 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:12.530606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"
Jul 23 09:18:12.662286 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:12.662255    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr"]
Jul 23 09:18:12.665223 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:12.665196    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1101cc3_0d14_489a_af1e_e38f6e3cbe67.slice/crio-aa02f1a19050f0352c5b83b9f901989e035712bffb47f363c44d23888457ac84 WatchSource:0}: Error finding container aa02f1a19050f0352c5b83b9f901989e035712bffb47f363c44d23888457ac84: Status 404 returned error can't find the container with id aa02f1a19050f0352c5b83b9f901989e035712bffb47f363c44d23888457ac84
Jul 23 09:18:13.246336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:13.246300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr" event={"ID":"f1101cc3-0d14-489a-af1e-e38f6e3cbe67","Type":"ContainerStarted","Data":"aa02f1a19050f0352c5b83b9f901989e035712bffb47f363c44d23888457ac84"}
Jul 23 09:18:14.251934 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:14.251887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr" event={"ID":"f1101cc3-0d14-489a-af1e-e38f6e3cbe67","Type":"ContainerStarted","Data":"d0d195a36d7736a74a7250e35319aa4ece77f37fbf00bd78903eaca3382451f1"}
Jul 23 09:18:14.252290 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:14.251961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr" event={"ID":"f1101cc3-0d14-489a-af1e-e38f6e3cbe67","Type":"ContainerStarted","Data":"fdb346110b3b2ea768b25bf038d3caec611a4ed60ddacf5d3d6c454ffa24c5b5"}
Jul 23 09:18:16.020416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.020366    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8lhnr" podStartSLOduration=3.905559607 podStartE2EDuration="5.020349585s" totalImagesPullingTime="1.114789978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:14.273941967 +0000 UTC m=+78.952159749" watchObservedRunningTime="2026-07-23 09:18:16.020349585 +0000 UTC m=+80.698567367"
Jul 23 09:18:16.020878 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.020862    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"]
Jul 23 09:18:16.023542 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.023527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.026488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.026465    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 23 09:18:16.026706 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.026679    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 23 09:18:16.026810 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.026787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8hbq2\""
Jul 23 09:18:16.038657 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.038632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"]
Jul 23 09:18:16.039526 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.039505    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"]
Jul 23 09:18:16.042034 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.042017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.045679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.045663    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 23 09:18:16.046247 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.046228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-27tnw\""
Jul 23 09:18:16.046468 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.046453    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 23 09:18:16.046873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.046846    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 23 09:18:16.068971 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.068951    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"]
Jul 23 09:18:16.069764 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.069747    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-25jw7"]
Jul 23 09:18:16.072664 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.072644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.075745 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.075733    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dcjpd\""
Jul 23 09:18:16.075837 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.075804    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 09:18:16.076078 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.076059    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 09:18:16.076260 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.076244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 09:18:16.113607 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.113701 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.113701 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cc311f4-b27d-48e0-b77e-d016f244e47f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.113701 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcwr\" (UniqueName: \"kubernetes.io/projected/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-kube-api-access-8dcwr\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.113701 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.113874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.113874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.113874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.113874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.113874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.113867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcsh\" (UniqueName: \"kubernetes.io/projected/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-api-access-rpcsh\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.214364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.214469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-metrics-client-ca\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.214469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.214469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-wtmp\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.214469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcsh\" (UniqueName: \"kubernetes.io/projected/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-api-access-rpcsh\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:16.214504    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:18:16.214558    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls podName:7cc22e99-9137-4fe1-90bb-a1a1fe741bc3 nodeName:}" failed. No retries permitted until 2026-07-23 09:18:16.714543507 +0000 UTC m=+81.392761268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-qjsnd" (UID: "7cc22e99-9137-4fe1-90bb-a1a1fe741bc3") : secret "openshift-state-metrics-tls" not found
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-textfile\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.214676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-root\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767cv\" (UniqueName: \"kubernetes.io/projected/a8244060-d400-443d-b19e-b02262c775a9-kube-api-access-767cv\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-tls\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cc311f4-b27d-48e0-b77e-d016f244e47f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcwr\" (UniqueName: \"kubernetes.io/projected/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-kube-api-access-8dcwr\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.214988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.215054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.215020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-sys\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.215505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.215212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.215505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.215383    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/5cc311f4-b27d-48e0-b77e-d016f244e47f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.215686 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.215660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.215771 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.215754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5cc311f4-b27d-48e0-b77e-d016f244e47f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.217085 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.217046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.217172 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.217116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.217213 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.217191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.225949 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.225910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcsh\" (UniqueName: \"kubernetes.io/projected/5cc311f4-b27d-48e0-b77e-d016f244e47f-kube-api-access-rpcsh\") pod \"kube-state-metrics-56c5594674-f7g8w\" (UID: \"5cc311f4-b27d-48e0-b77e-d016f244e47f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.226162 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.226145    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcwr\" (UniqueName: \"kubernetes.io/projected/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-kube-api-access-8dcwr\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.315406 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-metrics-client-ca\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315406 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315374    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-wtmp\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315406 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-textfile\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-root\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767cv\" (UniqueName: \"kubernetes.io/projected/a8244060-d400-443d-b19e-b02262c775a9-kube-api-access-767cv\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-tls\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-root\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-wtmp\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.315635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-sys\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.316008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8244060-d400-443d-b19e-b02262c775a9-sys\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.316008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.315791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-textfile\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.316092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.316029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-metrics-client-ca\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.316130 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.316100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-accelerators-collector-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.317718 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.317697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.317813 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.317800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8244060-d400-443d-b19e-b02262c775a9-node-exporter-tls\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.329291 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.329268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-767cv\" (UniqueName: \"kubernetes.io/projected/a8244060-d400-443d-b19e-b02262c775a9-kube-api-access-767cv\") pod \"node-exporter-25jw7\" (UID: \"a8244060-d400-443d-b19e-b02262c775a9\") " pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.351264 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.351245    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"
Jul 23 09:18:16.381102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.381083    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-25jw7"
Jul 23 09:18:16.393162 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:16.393138    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8244060_d400_443d_b19e_b02262c775a9.slice/crio-aa6a595a2afb8bfb5112af5ef12e0710cbdcc207da7963f5581c652503c292ab WatchSource:0}: Error finding container aa6a595a2afb8bfb5112af5ef12e0710cbdcc207da7963f5581c652503c292ab: Status 404 returned error can't find the container with id aa6a595a2afb8bfb5112af5ef12e0710cbdcc207da7963f5581c652503c292ab
Jul 23 09:18:16.483151 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.483124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-f7g8w"]
Jul 23 09:18:16.485403 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:16.485375    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc311f4_b27d_48e0_b77e_d016f244e47f.slice/crio-197d151201412ae4edf5b58d32da3f073d03a83d426fe5d8eee0499fa65879fe WatchSource:0}: Error finding container 197d151201412ae4edf5b58d32da3f073d03a83d426fe5d8eee0499fa65879fe: Status 404 returned error can't find the container with id 197d151201412ae4edf5b58d32da3f073d03a83d426fe5d8eee0499fa65879fe
Jul 23 09:18:16.718436 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.718408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.720558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.720532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc22e99-9137-4fe1-90bb-a1a1fe741bc3-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-qjsnd\" (UID: \"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:16.932382 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:16.932353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"
Jul 23 09:18:17.102175 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:17.102139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd"]
Jul 23 09:18:17.266474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:17.264165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25jw7" event={"ID":"a8244060-d400-443d-b19e-b02262c775a9","Type":"ContainerStarted","Data":"aa6a595a2afb8bfb5112af5ef12e0710cbdcc207da7963f5581c652503c292ab"}
Jul 23 09:18:17.266474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:17.266086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w" event={"ID":"5cc311f4-b27d-48e0-b77e-d016f244e47f","Type":"ContainerStarted","Data":"197d151201412ae4edf5b58d32da3f073d03a83d426fe5d8eee0499fa65879fe"}
Jul 23 09:18:17.300794 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:17.300759    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc22e99_9137_4fe1_90bb_a1a1fe741bc3.slice/crio-9c25b0f2ac9165d0908db601a28f9e5e8096e223447a60dcd33af287ec1a6a0d WatchSource:0}: Error finding container 9c25b0f2ac9165d0908db601a28f9e5e8096e223447a60dcd33af287ec1a6a0d: Status 404 returned error can't find the container with id 9c25b0f2ac9165d0908db601a28f9e5e8096e223447a60dcd33af287ec1a6a0d
Jul 23 09:18:18.273314 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.273247    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w" event={"ID":"5cc311f4-b27d-48e0-b77e-d016f244e47f","Type":"ContainerStarted","Data":"ec471d2cc618ff44a20d8a3932d7e34ba2e484dd1937bfba76b37acd86e90353"}
Jul 23 09:18:18.273789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.273329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w" event={"ID":"5cc311f4-b27d-48e0-b77e-d016f244e47f","Type":"ContainerStarted","Data":"92b6bb8fd6a7361bcd45586f62c8c55e74b7284a16995349716137be9a9e9133"}
Jul 23 09:18:18.273789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.273365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w" event={"ID":"5cc311f4-b27d-48e0-b77e-d016f244e47f","Type":"ContainerStarted","Data":"9e706e57f437f0ca091b52a04d00f634eb73c9c29d0eda7e41ff67dfa1af4f54"}
Jul 23 09:18:18.274994 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.274967    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8244060-d400-443d-b19e-b02262c775a9" containerID="af9c6e0dbe3edc2c523a5248150ace0b8fecab8eea60a68f0da5c93ed8a9acc4" exitCode=0
Jul 23 09:18:18.275117 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.275057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25jw7" event={"ID":"a8244060-d400-443d-b19e-b02262c775a9","Type":"ContainerDied","Data":"af9c6e0dbe3edc2c523a5248150ace0b8fecab8eea60a68f0da5c93ed8a9acc4"}
Jul 23 09:18:18.277399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.277374    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd" event={"ID":"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3","Type":"ContainerStarted","Data":"50e174bae5046020f538acb099e621f5a101f3c86a3dc67e7768c308ce0ae282"}
Jul 23 09:18:18.277486 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.277426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd" event={"ID":"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3","Type":"ContainerStarted","Data":"3d299c366c0a7e3bbc1055c8ca6dd1dfd99be640ca2a226949e8011bcda5760f"}
Jul 23 09:18:18.277486 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.277452    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd" event={"ID":"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3","Type":"ContainerStarted","Data":"9c25b0f2ac9165d0908db601a28f9e5e8096e223447a60dcd33af287ec1a6a0d"}
Jul 23 09:18:18.300013 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:18.299968    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-f7g8w" podStartSLOduration=0.891650604 podStartE2EDuration="2.299950004s" totalImagesPullingTime="1.4082994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:18.299141753 +0000 UTC m=+82.977359573" watchObservedRunningTime="2026-07-23 09:18:18.299950004 +0000 UTC m=+82.978167806"
Jul 23 09:18:19.062831 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.062791    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84c4756556-wcnzv"]
Jul 23 09:18:19.066442 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.066425    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.069383 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.069360    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 23 09:18:19.069383 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.069372    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 23 09:18:19.069552 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.069362    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 23 09:18:19.069658 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.069645    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 23 09:18:19.069861 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.069846    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c8a7ae0jcnsbt\""
Jul 23 09:18:19.070275 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.070256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ffbwg\""
Jul 23 09:18:19.070365 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.070260    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 23 09:18:19.083235 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.083217    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84c4756556-wcnzv"]
Jul 23 09:18:19.137826 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.137800    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.137910 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.137833    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.137910 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.137900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0bc11-d4b4-416c-a62e-c45aa1463576-metrics-client-ca\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.138089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.137947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbhp\" (UniqueName: \"kubernetes.io/projected/e7f0bc11-d4b4-416c-a62e-c45aa1463576-kube-api-access-lbbhp\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.138089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.138026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.138171 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.138082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.138171 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.138140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.138245 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.138170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-grpc-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.232231 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.232206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cs9ms"
Jul 23 09:18:19.238627 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0bc11-d4b4-416c-a62e-c45aa1463576-metrics-client-ca\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238716 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbhp\" (UniqueName: \"kubernetes.io/projected/e7f0bc11-d4b4-416c-a62e-c45aa1463576-kube-api-access-lbbhp\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238716 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238802 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238841 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238885 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-grpc-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238953 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.238953 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.238933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.239567 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.239546    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7f0bc11-d4b4-416c-a62e-c45aa1463576-metrics-client-ca\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.241488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.241469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.241562 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.241536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-grpc-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.241908 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.241873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.242008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.241993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-tls\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.242071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.242028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.242123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.242102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e7f0bc11-d4b4-416c-a62e-c45aa1463576-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.249372 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.249346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbhp\" (UniqueName: \"kubernetes.io/projected/e7f0bc11-d4b4-416c-a62e-c45aa1463576-kube-api-access-lbbhp\") pod \"thanos-querier-84c4756556-wcnzv\" (UID: \"e7f0bc11-d4b4-416c-a62e-c45aa1463576\") " pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.284217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.284159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25jw7" event={"ID":"a8244060-d400-443d-b19e-b02262c775a9","Type":"ContainerStarted","Data":"01a63e40518ae493e3013c1199d07b5f9889a2a5afb4d07cb1537e1959133c86"}
Jul 23 09:18:19.284981 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.284881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-25jw7" event={"ID":"a8244060-d400-443d-b19e-b02262c775a9","Type":"ContainerStarted","Data":"d70b874df3eea44778132cf1eabbae5982543763c82a56c9454f26f85cc56f4f"}
Jul 23 09:18:19.288192 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.288169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd" event={"ID":"7cc22e99-9137-4fe1-90bb-a1a1fe741bc3","Type":"ContainerStarted","Data":"429d32d1c4933c3586d15293f73fbeb79291a3c565c8f176c8d493b61b172c41"}
Jul 23 09:18:19.317330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.317256    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-25jw7" podStartSLOduration=2.290917854 podStartE2EDuration="3.317243568s" totalImagesPullingTime="955.969857ms" totalInitContainerRuntime="70.355857ms" isStatefulPod=true podCreationTimestamp="2026-07-23 09:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:19.31637251 +0000 UTC m=+83.994590292" watchObservedRunningTime="2026-07-23 09:18:19.317243568 +0000 UTC m=+83.995461348"
Jul 23 09:18:19.340724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.340680    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-qjsnd" podStartSLOduration=3.130794178 podStartE2EDuration="4.340668076s" totalImagesPullingTime="1.209873898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:19.339729811 +0000 UTC m=+84.017947626" watchObservedRunningTime="2026-07-23 09:18:19.340668076 +0000 UTC m=+84.018885859"
Jul 23 09:18:19.375463 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.375440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:19.510356 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:19.510334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84c4756556-wcnzv"]
Jul 23 09:18:19.512582 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:19.512555    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f0bc11_d4b4_416c_a62e_c45aa1463576.slice/crio-beb77ba4602000faa21dfe397bef5e2def6098f5d68f8d4dbbed776c5a4ba236 WatchSource:0}: Error finding container beb77ba4602000faa21dfe397bef5e2def6098f5d68f8d4dbbed776c5a4ba236: Status 404 returned error can't find the container with id beb77ba4602000faa21dfe397bef5e2def6098f5d68f8d4dbbed776c5a4ba236
Jul 23 09:18:20.294065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.294018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"beb77ba4602000faa21dfe397bef5e2def6098f5d68f8d4dbbed776c5a4ba236"}
Jul 23 09:18:20.784214 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.784178    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"]
Jul 23 09:18:20.787511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.787478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.790203 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.790178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 09:18:20.790334 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.790252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4vnmz\""
Jul 23 09:18:20.797410 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.797389    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"]
Jul 23 09:18:20.852553 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.852528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.852680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.852569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wcpb\" (UniqueName: \"kubernetes.io/projected/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-kube-api-access-8wcpb\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.953283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.953258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.953454 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.953301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wcpb\" (UniqueName: \"kubernetes.io/projected/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-kube-api-access-8wcpb\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.956235 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.956211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:20.965622 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:20.965591    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wcpb\" (UniqueName: \"kubernetes.io/projected/c3e86f29-9b75-4f4c-a849-f8fe32f7e83b-kube-api-access-8wcpb\") pod \"monitoring-plugin-7b7b7bc8bf-j8dnb\" (UID: \"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:21.098423 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:21.098349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:21.675181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:21.675155    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"]
Jul 23 09:18:21.678040 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:21.678013    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e86f29_9b75_4f4c_a849_f8fe32f7e83b.slice/crio-e432d770350c9d6c6c6cacfd33b7c94960b805b6b8cffa157d952ba8e24e7002 WatchSource:0}: Error finding container e432d770350c9d6c6c6cacfd33b7c94960b805b6b8cffa157d952ba8e24e7002: Status 404 returned error can't find the container with id e432d770350c9d6c6c6cacfd33b7c94960b805b6b8cffa157d952ba8e24e7002
Jul 23 09:18:22.308175 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.308094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb" event={"ID":"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b","Type":"ContainerStarted","Data":"e432d770350c9d6c6c6cacfd33b7c94960b805b6b8cffa157d952ba8e24e7002"}
Jul 23 09:18:22.312989 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.312802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"155d04c401906e8b74bbc8135469cf133cd9141ce276d58f746af0bb76cc3150"}
Jul 23 09:18:22.312989 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.312876    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"7c06d640b526e2d8ac905053db029e679fbd1ffb2bfe3666ecf6d84fcc0c1231"}
Jul 23 09:18:22.312989 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.312943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"93a39b8ed836064f30a36d61dddb43317e4eca31a219aa78b6ab6f617b065290"}
Jul 23 09:18:22.347795 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.347330    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:18:22.353281 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.353256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.357222 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.357198    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 09:18:22.357961 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.357792    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.358272    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.358301    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.358570    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9998w\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.358730    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.359012    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.359234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 09:18:22.359528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.359464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 09:18:22.359941 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.359553    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 09:18:22.360283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.360261    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 09:18:22.364360 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.364338    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 09:18:22.364591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.364535    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 09:18:22.364591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.364564    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fhfm7arnfuc4e\""
Jul 23 09:18:22.364740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.364672    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 09:18:22.366808 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.366790    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 09:18:22.368666 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.368647    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 09:18:22.377891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.377857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:18:22.464974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.464899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.464974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.464964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465054    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9h9n\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465249    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465531    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.465887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.465720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.567959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.568330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.567977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.568330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9h9n\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.568330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.568330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.568330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.568713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.569500 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.569126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.569775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.569817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.570290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.571396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.572033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.573336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.572965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.576183 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.576144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.576362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.576336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.578308 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.578286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.579188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.579087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.579188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.579087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.579335 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.579317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.579571 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.579521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.580402 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.579962    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.580402 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.580370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.580997 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.580682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.580997 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.580960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.581126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.581004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.583725 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.583686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9h9n\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n\") pod \"prometheus-k8s-0\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.670415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.670373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:22.842365 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:22.842340    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:18:22.845674 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:18:22.845580    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc4b076_fcfe_41d2_9a60_30647533c1d8.slice/crio-7f75ce006639f2f8723ba126d7ee329542f7534399aff00a376c87d5d3d009ca WatchSource:0}: Error finding container 7f75ce006639f2f8723ba126d7ee329542f7534399aff00a376c87d5d3d009ca: Status 404 returned error can't find the container with id 7f75ce006639f2f8723ba126d7ee329542f7534399aff00a376c87d5d3d009ca
Jul 23 09:18:23.321612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.321515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"cc2c983576eb2580bf69934876d13491b2d7ae885c940016fcf49a3981f7406b"}
Jul 23 09:18:23.321612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.321596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"de1ef8efccaa53822800c8462e521ec7144b6647befd49eb3db4c5bc39fbf712"}
Jul 23 09:18:23.321766 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.321638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" event={"ID":"e7f0bc11-d4b4-416c-a62e-c45aa1463576","Type":"ContainerStarted","Data":"6bc2800368fadc6b24644f195f7d86cb7113f9e6e5879f6266df3600c772b024"}
Jul 23 09:18:23.321766 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.321684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:23.323121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.323096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"7f75ce006639f2f8723ba126d7ee329542f7534399aff00a376c87d5d3d009ca"}
Jul 23 09:18:23.354769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:23.354705    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv" podStartSLOduration=1.370601827 podStartE2EDuration="4.354680503s" totalImagesPullingTime="2.984078676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:23.351956829 +0000 UTC m=+88.030174634" watchObservedRunningTime="2026-07-23 09:18:23.354680503 +0000 UTC m=+88.032898286"
Jul 23 09:18:24.327842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.327807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb" event={"ID":"c3e86f29-9b75-4f4c-a849-f8fe32f7e83b","Type":"ContainerStarted","Data":"19ae5b91d6b54649a6e920ded9bb4d78ee625b5376530e220ccab4b6bd8c06c3"}
Jul 23 09:18:24.328292 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.327877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:24.329414 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.329392    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178" exitCode=0
Jul 23 09:18:24.329553 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.329470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"}
Jul 23 09:18:24.333424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.333406    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb"
Jul 23 09:18:24.354895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:24.354856    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-j8dnb" podStartSLOduration=2.760432948 podStartE2EDuration="4.354843001s" totalImagesPullingTime="1.594410053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:24.353255393 +0000 UTC m=+89.031473175" watchObservedRunningTime="2026-07-23 09:18:24.354843001 +0000 UTC m=+89.033060824"
Jul 23 09:18:25.206763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:25.206733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c6c89897c-2zg9x"
Jul 23 09:18:28.352539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"}
Jul 23 09:18:28.353043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"}
Jul 23 09:18:28.353043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"}
Jul 23 09:18:28.353043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"}
Jul 23 09:18:28.353043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"}
Jul 23 09:18:28.353043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.352694    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerStarted","Data":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"}
Jul 23 09:18:28.384126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:28.384071    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.0939397 podStartE2EDuration="6.384051051s" totalImagesPullingTime="4.272642689s" totalInitContainerRuntime="17.468662ms" isStatefulPod=false podCreationTimestamp="2026-07-23 09:18:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:18:28.38165238 +0000 UTC m=+93.059870183" watchObservedRunningTime="2026-07-23 09:18:28.384051051 +0000 UTC m=+93.062268835"
Jul 23 09:18:29.336240 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:29.336215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84c4756556-wcnzv"
Jul 23 09:18:32.671459 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:32.671423    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:18:40.231894 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:18:40.231865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nmlww"
Jul 23 09:19:22.671466 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:22.671428    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:22.691512 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:22.691489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:23.573149 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:23.573123    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:40.997483 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997429    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997777    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="prometheus" containerID="cri-o://ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512" gracePeriod=600
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997819    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="config-reloader" containerID="cri-o://17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e" gracePeriod=600
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997799    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="thanos-sidecar" containerID="cri-o://68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f" gracePeriod=600
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997792    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-web" containerID="cri-o://ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a" gracePeriod=600
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997796    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162" gracePeriod=600
Jul 23 09:19:40.998065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:40.997820    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy" containerID="cri-o://40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda" gracePeriod=600
Jul 23 09:19:41.246086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.246057    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.380578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380528    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380548    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380581    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380735    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380808    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.380842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380839    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380878    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9h9n\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380885    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380910    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380946    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380959    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.380995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381029    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381060    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381126 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381149    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs\") pod \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\" (UID: \"1dc4b076-fcfe-41d2-9a60-30647533c1d8\") "
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381518    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-metrics-client-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.381626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381537    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.383484 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381051    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.383605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.383537    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.383605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.381122    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.383605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.382121    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.383605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.383300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n" (OuterVolumeSpecName: "kube-api-access-l9h9n") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "kube-api-access-l9h9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:19:41.383605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.383442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.383906 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.383872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:19:41.384140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.384115    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out" (OuterVolumeSpecName: "config-out") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:19:41.384403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.384371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:19:41.384740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.384708    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.384895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.384869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.385036 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.384995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.385163 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.385142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:19:41.385475 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.385457    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.385582 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.385558    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.386070 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.386052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config" (OuterVolumeSpecName: "config") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.386602 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.386585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.394355 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.394337    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config" (OuterVolumeSpecName: "web-config") pod "1dc4b076-fcfe-41d2-9a60-30647533c1d8" (UID: "1dc4b076-fcfe-41d2-9a60-30647533c1d8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:19:41.482884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482860    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.482884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482882    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-db\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482893    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482902    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-web-config\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482931    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9h9n\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-kube-api-access-l9h9n\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482940    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482949    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482958    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-grpc-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482966    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-kube-rbac-proxy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482974    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482983    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc4b076-fcfe-41d2-9a60-30647533c1d8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482991    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.482999    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.483010    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc4b076-fcfe-41d2-9a60-30647533c1d8-tls-assets\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.483017    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.483026    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-secret-metrics-client-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.483033    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc4b076-fcfe-41d2-9a60-30647533c1d8-config-out\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.483038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.483042    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1dc4b076-fcfe-41d2-9a60-30647533c1d8-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:19:41.626426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626396    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162" exitCode=0
Jul 23 09:19:41.626426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626423    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda" exitCode=0
Jul 23 09:19:41.626426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626429    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a" exitCode=0
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626437    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f" exitCode=0
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626443    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e" exitCode=0
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626448    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512" exitCode=0
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626471    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"}
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"}
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"}
Jul 23 09:19:41.626612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"}
Jul 23 09:19:41.626986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"}
Jul 23 09:19:41.626986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626648    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.626986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"}
Jul 23 09:19:41.626986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.626699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1dc4b076-fcfe-41d2-9a60-30647533c1d8","Type":"ContainerDied","Data":"7f75ce006639f2f8723ba126d7ee329542f7534399aff00a376c87d5d3d009ca"}
Jul 23 09:19:41.634440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.634423    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.641083 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.641067    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.647699 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.647681    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.654100 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.654084    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.660489 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.660472    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.666992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.666978    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.673604 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.673589    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.673840 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.673823    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.673884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.673847    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.673884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.673875    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.674131 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.674112    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.674204 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674138    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.674204 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674160    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.674428 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.674412    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.674472 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674432    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.674472 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674445    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.674707 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.674687    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.674751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.674751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.674727    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.675238 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.675219    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.675330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675255    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.675330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675275    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.675570 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.675546    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.675654 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675567    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.675654 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675580    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.675847 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:19:41.675818    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.675970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675852    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.675970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.675867    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.676090 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676078    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.676148 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676092    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.676307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676287    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.676366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676307    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.676501 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676481    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.676554 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676501    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.676710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676687    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.676787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676707    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:41.676787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.676710    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.677038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677020    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.677094 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677039    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.677250 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677234    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.677287 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677250    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.677461 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677446    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.677498 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677462    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.677646 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677631    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.677684 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677648    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.677837 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677822    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.677880 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.677837    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.678040 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678014    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.678040 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678040    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.678270 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678252    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.678343 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678272    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.678511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678494    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.678560 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678512    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.678723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678702    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.678723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678722    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.678948 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678908    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.679051 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.678950    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.679237 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679213    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.679237 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679237    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.679518 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679498    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.679612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679520    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.679814 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679796    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.679895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.679817    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.680086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680065    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.680157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680093    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.680550 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680521    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.680550 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680544    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.680805 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680786    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.680863 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.680807    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.681054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681037    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.681102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681056    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.681280 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681262    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.681327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681281    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.681510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681494    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.681510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681509    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.681709 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681691    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.681769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681710    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.681970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681900    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.681970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.681949    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.682173 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682156    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:41.682173 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682166    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.682253 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682181    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.682405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682387    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.682475 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682407    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.682641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682624    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.682698 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682642    2560 scope.go:117] "RemoveContainer" containerID="e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"
Jul 23 09:19:41.682865 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682848    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162"} err="failed to get container status \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": rpc error: code = NotFound desc = could not find container \"e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162\": container with ID starting with e230bb7d3bfee1182a36ff071fa312519ac75033f412abefdea53dbf6334e162 not found: ID does not exist"
Jul 23 09:19:41.682924 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.682866    2560 scope.go:117] "RemoveContainer" containerID="40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"
Jul 23 09:19:41.683094 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683073    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda"} err="failed to get container status \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": rpc error: code = NotFound desc = could not find container \"40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda\": container with ID starting with 40c01eb168686c79b06b0ad3746b14ac67b1c533a44d0fec5d0c558b0936aeda not found: ID does not exist"
Jul 23 09:19:41.683139 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683095    2560 scope.go:117] "RemoveContainer" containerID="ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"
Jul 23 09:19:41.683313 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683290    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a"} err="failed to get container status \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": rpc error: code = NotFound desc = could not find container \"ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a\": container with ID starting with ac4bb0b8c105dc6329966e6aa76959e6d6a6e2e9d01f13f4e821153e9576473a not found: ID does not exist"
Jul 23 09:19:41.683367 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683314    2560 scope.go:117] "RemoveContainer" containerID="68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"
Jul 23 09:19:41.683509 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683492    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f"} err="failed to get container status \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": rpc error: code = NotFound desc = could not find container \"68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f\": container with ID starting with 68694d4bd6ea9f7e022d192195758acf8cab41beb1d17e35536437dcfde8421f not found: ID does not exist"
Jul 23 09:19:41.683549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683510    2560 scope.go:117] "RemoveContainer" containerID="17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"
Jul 23 09:19:41.683734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683719    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e"} err="failed to get container status \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": rpc error: code = NotFound desc = could not find container \"17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e\": container with ID starting with 17a304e7fbf26662decab2e4a4a9e029d0b88e3acfd4404ef5cff2123546906e not found: ID does not exist"
Jul 23 09:19:41.683734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683733    2560 scope.go:117] "RemoveContainer" containerID="ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"
Jul 23 09:19:41.683959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683939    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512"} err="failed to get container status \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": rpc error: code = NotFound desc = could not find container \"ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512\": container with ID starting with ef8e91f8f9f7c1cf68fb664b845c49041bcfb54e2537aad42fcac768af87f512 not found: ID does not exist"
Jul 23 09:19:41.684035 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.683961    2560 scope.go:117] "RemoveContainer" containerID="7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"
Jul 23 09:19:41.684172 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.684155    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178"} err="failed to get container status \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": rpc error: code = NotFound desc = could not find container \"7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178\": container with ID starting with 7ae78e9713db34dcac823feafbd16094580394a5f639931cf4add0d5ecaf1178 not found: ID does not exist"
Jul 23 09:19:41.761224 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761202    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:41.761467 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761456    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="init-config-reloader"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761468    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="init-config-reloader"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761479    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="config-reloader"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761485    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="config-reloader"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761495    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-thanos"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761500    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-thanos"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761509    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="prometheus"
Jul 23 09:19:41.761513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761514    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="prometheus"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761523    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-web"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761528    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-web"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761538    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="thanos-sidecar"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761543    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="thanos-sidecar"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761548    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761554    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761594    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761601    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="config-reloader"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761610    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="prometheus"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761617    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="thanos-sidecar"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761622    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-web"
Jul 23 09:19:41.761705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.761626    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" containerName="kube-rbac-proxy-thanos"
Jul 23 09:19:41.766270 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.766256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.768998 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.768981    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 09:19:41.769195 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.769182    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fhfm7arnfuc4e\""
Jul 23 09:19:41.769438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.769416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 09:19:41.769528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.769458    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 09:19:41.769528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.769507    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 09:19:41.770700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.770683    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 09:19:41.770951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.770935    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 09:19:41.771017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.770980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 09:19:41.771261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.771085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 09:19:41.771261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.771100    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 09:19:41.771261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.771118    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 09:19:41.771261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.771138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 09:19:41.771474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.771461    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 09:19:41.772552 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.772531    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9998w\""
Jul 23 09:19:41.772728 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.772707    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 09:19:41.774106 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.774089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 09:19:41.775532 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.775513    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 09:19:41.794750 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.794726    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:41.886937 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.886868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.886937 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.886898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.886937 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.886931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.886948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-web-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbr6p\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-kube-api-access-dbr6p\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.887491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.887468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-config-out\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.911596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.911575    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc4b076-fcfe-41d2-9a60-30647533c1d8" path="/var/lib/kubelet/pods/1dc4b076-fcfe-41d2-9a60-30647533c1d8/volumes"
Jul 23 09:19:41.988413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbr6p\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-kube-api-access-dbr6p\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-config-out\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988612    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988695    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-web-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.988987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.988849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.989187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.989956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.990002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.990182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.990645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.991013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.991227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.991965 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.991942    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/86d292c4-c36f-4855-b678-397d99952902-config-out\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/86d292c4-c36f-4855-b678-397d99952902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.993858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.994216 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.993858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.994216 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.994030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.994216 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.994116    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:41.994542 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:41.994526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/86d292c4-c36f-4855-b678-397d99952902-web-config\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:42.000047 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.000025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbr6p\" (UniqueName: \"kubernetes.io/projected/86d292c4-c36f-4855-b678-397d99952902-kube-api-access-dbr6p\") pod \"prometheus-k8s-0\" (UID: \"86d292c4-c36f-4855-b678-397d99952902\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:42.076539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.076517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:19:42.236353 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.236323    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 09:19:42.246563 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:19:42.246535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d292c4_c36f_4855_b678_397d99952902.slice/crio-c83aa8c6049a76b6e65c6d01fe3a430196d5f63e0b98f59f6d673898d0d6c20c WatchSource:0}: Error finding container c83aa8c6049a76b6e65c6d01fe3a430196d5f63e0b98f59f6d673898d0d6c20c: Status 404 returned error can't find the container with id c83aa8c6049a76b6e65c6d01fe3a430196d5f63e0b98f59f6d673898d0d6c20c
Jul 23 09:19:42.633164 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.633128    2560 generic.go:356] "Generic (PLEG): container finished" podID="86d292c4-c36f-4855-b678-397d99952902" containerID="fd0c6d33f5c1738f5ba6099ba3868cf2fa58e5ecf80c6e02a1c7e0d79c982474" exitCode=0
Jul 23 09:19:42.633296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.633214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerDied","Data":"fd0c6d33f5c1738f5ba6099ba3868cf2fa58e5ecf80c6e02a1c7e0d79c982474"}
Jul 23 09:19:42.633373 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:42.633330    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"c83aa8c6049a76b6e65c6d01fe3a430196d5f63e0b98f59f6d673898d0d6c20c"}
Jul 23 09:19:43.643698 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"9bfeadc3bc9f968595e698460596952dc1ed5ac99f0a58a328775d015a1a99e8"}
Jul 23 09:19:43.644108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"da1afd2994ccd74d0cbbf91fdf18394de5ac359f511cfc112e1121c915a799f5"}
Jul 23 09:19:43.644108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643760    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"3a7e296d36124c77e54401943a3f48d1c0445fae141bb2c48b0b283b3c9ef9c6"}
Jul 23 09:19:43.644108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643785    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"51d5a0c8eb833cf9ff4fe9ddb280800cffa1b4a601b4d000b61f22df0c7e2bfa"}
Jul 23 09:19:43.644108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"3540c357ba2b2df8ad2cab6ec92821a05fca1b70cb0939a0b16677c8692e2ac2"}
Jul 23 09:19:43.644108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.643839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"86d292c4-c36f-4855-b678-397d99952902","Type":"ContainerStarted","Data":"d829b5431660c7953041aa6fcadaace51f749d172d3a8ff642127eb929f3e941"}
Jul 23 09:19:43.687136 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:43.687087    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.667364285 podStartE2EDuration="2.687070929s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.706644ms" isStatefulPod=false podCreationTimestamp="2026-07-23 09:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:19:43.685164217 +0000 UTC m=+168.363382074" watchObservedRunningTime="2026-07-23 09:19:43.687070929 +0000 UTC m=+168.365288712"
Jul 23 09:19:47.076806 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:19:47.076769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:20:42.076874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:20:42.076835    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:20:42.091801 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:20:42.091777    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:20:42.886082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:20:42.886056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 09:21:54.157801 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.157772    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"]
Jul 23 09:21:54.161091 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.161071    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.166005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.164879    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 09:21:54.166005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.165728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lfn6j\""
Jul 23 09:21:54.166418 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.166400    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 09:21:54.166628 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.166612    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 09:21:54.167335 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.167312    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 09:21:54.187700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.187666    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"]
Jul 23 09:21:54.245243 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.245216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.245376 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.245246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.245376 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.245266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.245376 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.245337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chnt\" (UniqueName: \"kubernetes.io/projected/7051f641-18e2-44e6-80e3-b581c48b5066-kube-api-access-7chnt\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.245504 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.245376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chnt\" (UniqueName: \"kubernetes.io/projected/7051f641-18e2-44e6-80e3-b581c48b5066-kube-api-access-7chnt\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.346842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.346818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-manifests\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.348734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.348706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7051f641-18e2-44e6-80e3-b581c48b5066-tmp\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.348862 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.348843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.348910 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.348847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7051f641-18e2-44e6-80e3-b581c48b5066-webhook-cert\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.354834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.354812    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chnt\" (UniqueName: \"kubernetes.io/projected/7051f641-18e2-44e6-80e3-b581c48b5066-kube-api-access-7chnt\") pod \"opendatahub-operator-controller-manager-64d6699bc9-2765v\" (UID: \"7051f641-18e2-44e6-80e3-b581c48b5066\") " pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.475352 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.475249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:21:54.614170 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:54.614139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"]
Jul 23 09:21:54.617314 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:21:54.617286    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7051f641_18e2_44e6_80e3_b581c48b5066.slice/crio-66b02be2a92a8aaa5e5bf23af9a7684e3ab4b41b166e3244068016cfb5eb4b91 WatchSource:0}: Error finding container 66b02be2a92a8aaa5e5bf23af9a7684e3ab4b41b166e3244068016cfb5eb4b91: Status 404 returned error can't find the container with id 66b02be2a92a8aaa5e5bf23af9a7684e3ab4b41b166e3244068016cfb5eb4b91
Jul 23 09:21:55.128322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:55.128282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v" event={"ID":"7051f641-18e2-44e6-80e3-b581c48b5066","Type":"ContainerStarted","Data":"66b02be2a92a8aaa5e5bf23af9a7684e3ab4b41b166e3244068016cfb5eb4b91"}
Jul 23 09:21:55.756828 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:55.756793    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 09:21:58.143980 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:58.143941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v" event={"ID":"7051f641-18e2-44e6-80e3-b581c48b5066","Type":"ContainerStarted","Data":"b39d68b9c55513ea02af429fdc04b11f8547e9a56aeabfe9b82f8342f62c8968"}
Jul 23 09:21:59.149255 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:59.149162    2560 generic.go:356] "Generic (PLEG): container finished" podID="7051f641-18e2-44e6-80e3-b581c48b5066" containerID="b39d68b9c55513ea02af429fdc04b11f8547e9a56aeabfe9b82f8342f62c8968" exitCode=0
Jul 23 09:21:59.149255 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:59.149215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v" event={"ID":"7051f641-18e2-44e6-80e3-b581c48b5066","Type":"ContainerDied","Data":"b39d68b9c55513ea02af429fdc04b11f8547e9a56aeabfe9b82f8342f62c8968"}
Jul 23 09:21:59.149717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:21:59.149699    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:22:00.155403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:00.155364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v" event={"ID":"7051f641-18e2-44e6-80e3-b581c48b5066","Type":"ContainerStarted","Data":"44361aa18c7290198ff2ad100d9e31a04f569bd1a09b79dcc0c90bda648fb0d3"}
Jul 23 09:22:00.155820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:00.155529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:22:00.183881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:00.183836    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v" podStartSLOduration=1.735476233 podStartE2EDuration="6.183822388s" totalImagesPullingTime="3.39053402s" totalInitContainerRuntime="1.057812135s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:21:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:00.181232075 +0000 UTC m=+304.859449860" watchObservedRunningTime="2026-07-23 09:22:00.183822388 +0000 UTC m=+304.862040230"
Jul 23 09:22:11.162037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:11.162001    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64d6699bc9-2765v"
Jul 23 09:22:14.637430 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.635136    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8vjr4"]
Jul 23 09:22:14.648900 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.648873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8vjr4"]
Jul 23 09:22:14.649061 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.648987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.652115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.652075    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nct74\""
Jul 23 09:22:14.652267 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.652196    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 23 09:22:14.810831 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.810794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/bd2e00f1-3288-43a1-b3ee-c11dd12442db-kube-api-access-cwsdl\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.811011 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.810861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd2e00f1-3288-43a1-b3ee-c11dd12442db-tls-certs\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.911313 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.911268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/bd2e00f1-3288-43a1-b3ee-c11dd12442db-kube-api-access-cwsdl\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.911484 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.911358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd2e00f1-3288-43a1-b3ee-c11dd12442db-tls-certs\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.913787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.913760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd2e00f1-3288-43a1-b3ee-c11dd12442db-tls-certs\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.919678 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.919650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/bd2e00f1-3288-43a1-b3ee-c11dd12442db-kube-api-access-cwsdl\") pod \"model-serving-api-54d89bd5fc-8vjr4\" (UID: \"bd2e00f1-3288-43a1-b3ee-c11dd12442db\") " pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:14.960570 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:14.960542    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:15.122810 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:15.122771    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-8vjr4"]
Jul 23 09:22:15.125541 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:22:15.125506    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2e00f1_3288_43a1_b3ee_c11dd12442db.slice/crio-de382b6b5478f83bd6775adbcba702d2a35f52ef741e9bcaee8482c71e7177a3 WatchSource:0}: Error finding container de382b6b5478f83bd6775adbcba702d2a35f52ef741e9bcaee8482c71e7177a3: Status 404 returned error can't find the container with id de382b6b5478f83bd6775adbcba702d2a35f52ef741e9bcaee8482c71e7177a3
Jul 23 09:22:15.215873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:15.215774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4" event={"ID":"bd2e00f1-3288-43a1-b3ee-c11dd12442db","Type":"ContainerStarted","Data":"de382b6b5478f83bd6775adbcba702d2a35f52ef741e9bcaee8482c71e7177a3"}
Jul 23 09:22:18.231414 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:18.231373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4" event={"ID":"bd2e00f1-3288-43a1-b3ee-c11dd12442db","Type":"ContainerStarted","Data":"fbea8e113937aebfad405bc14af71d9795bb616bc987cea87574103f8473fd3b"}
Jul 23 09:22:18.232012 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:18.231437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:18.249812 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:18.249754    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4" podStartSLOduration=2.014864856 podStartE2EDuration="4.249734084s" totalImagesPullingTime="2.234869228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:18.24753665 +0000 UTC m=+322.925754467" watchObservedRunningTime="2026-07-23 09:22:18.249734084 +0000 UTC m=+322.927951867"
Jul 23 09:22:20.631946 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.631899    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:20.635269 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.635246    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.646905 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.646884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 09:22:20.647149 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.647135    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:22:20.648393 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.648380    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9f95b\""
Jul 23 09:22:20.648443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.648385    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 09:22:20.648443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.648413    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 09:22:20.653082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.653069    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 09:22:20.661164 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.661134    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 09:22:20.687324 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.687304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:20.761910 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.761887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762025 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.761942    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762025 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.762018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.762042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.762069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.762088    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr5s\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.762188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.762143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863068 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863429 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr5s\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863429 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.863866 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.863839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.865673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.865649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.865834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.865708    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.865834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.865714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.865990 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.865841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.873456 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.873420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr5s\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.874573 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.874546    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token\") pod \"istiod-openshift-gateway-7574459656-klfwz\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:20.943947 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:20.943908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:21.094170 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:21.094147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:21.096058 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:22:21.096030    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bac073d_efd8_4c16_8e39_3af677f3ad8e.slice/crio-c3915a548e6e576704b6e011747a90052312007013e587c5e35938ff344755b6 WatchSource:0}: Error finding container c3915a548e6e576704b6e011747a90052312007013e587c5e35938ff344755b6: Status 404 returned error can't find the container with id c3915a548e6e576704b6e011747a90052312007013e587c5e35938ff344755b6
Jul 23 09:22:21.245458 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:21.245375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" event={"ID":"3bac073d-efd8-4c16-8e39-3af677f3ad8e","Type":"ContainerStarted","Data":"c3915a548e6e576704b6e011747a90052312007013e587c5e35938ff344755b6"}
Jul 23 09:22:22.495854 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.495810    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"]
Jul 23 09:22:22.499424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.499402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.502393 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.502370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 09:22:22.503303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.503281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2mvw2\""
Jul 23 09:22:22.503448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.503431    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 09:22:22.503546 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.503510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 09:22:22.503546 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.503513    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 09:22:22.504097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.504081    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 09:22:22.513313 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.513287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"]
Jul 23 09:22:22.677641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.677603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.677641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.677640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jztcz\" (UniqueName: \"kubernetes.io/projected/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-kube-api-access-jztcz\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.677834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.677679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.677834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.677699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-manager-config\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.778721 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.778639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.778721 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.778696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-manager-config\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.779034 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.778782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.779034 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.778821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jztcz\" (UniqueName: \"kubernetes.io/projected/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-kube-api-access-jztcz\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.779488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.779461    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-manager-config\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.781314 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.781285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.781412 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.781363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-metrics-cert\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.793998 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.793970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jztcz\" (UniqueName: \"kubernetes.io/projected/3cfe5411-dc5f-4f5a-a663-c96a96515a5a-kube-api-access-jztcz\") pod \"lws-controller-manager-6cb78dd96b-hrk74\" (UID: \"3cfe5411-dc5f-4f5a-a663-c96a96515a5a\") " pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:22.810814 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:22.810787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:23.584118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:23.584075    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:22:23.584515 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:23.584148    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:22:23.687346 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:23.687318    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"]
Jul 23 09:22:23.690012 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:22:23.689986    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfe5411_dc5f_4f5a_a663_c96a96515a5a.slice/crio-24767c5a4f7def5675c7706bd7a95f0a45f3bc9b63c46867ae3c768e4711e6ee WatchSource:0}: Error finding container 24767c5a4f7def5675c7706bd7a95f0a45f3bc9b63c46867ae3c768e4711e6ee: Status 404 returned error can't find the container with id 24767c5a4f7def5675c7706bd7a95f0a45f3bc9b63c46867ae3c768e4711e6ee
Jul 23 09:22:24.272705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.271064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" event={"ID":"3bac073d-efd8-4c16-8e39-3af677f3ad8e","Type":"ContainerStarted","Data":"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"}
Jul 23 09:22:24.272705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.272083    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:24.273825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.273793    2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-klfwz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 09:22:24.273969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.273849    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 09:22:24.274266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.274242    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74" event={"ID":"3cfe5411-dc5f-4f5a-a663-c96a96515a5a","Type":"ContainerStarted","Data":"24767c5a4f7def5675c7706bd7a95f0a45f3bc9b63c46867ae3c768e4711e6ee"}
Jul 23 09:22:24.330510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:24.330394    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" podStartSLOduration=1.84492994 podStartE2EDuration="4.330374852s" totalImagesPullingTime="2.485444912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:24.327725318 +0000 UTC m=+329.005943129" watchObservedRunningTime="2026-07-23 09:22:24.330374852 +0000 UTC m=+329.008592636"
Jul 23 09:22:25.279527 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:25.279491    2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-klfwz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 09:22:25.279940 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:25.279576    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 09:22:26.285774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:26.285693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74" event={"ID":"3cfe5411-dc5f-4f5a-a663-c96a96515a5a","Type":"ContainerStarted","Data":"b47d936f2d92ff75f520a4b471a8bb9330e12b8ce25ab17b9b9f1a9fe8aa6877"}
Jul 23 09:22:26.286199 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:26.286060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:22:26.286323 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:26.286307    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:26.312904 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:26.312851    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74" podStartSLOduration=2.006771726 podStartE2EDuration="4.312832434s" totalImagesPullingTime="2.306060708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:22:26.3105818 +0000 UTC m=+330.988799628" watchObservedRunningTime="2026-07-23 09:22:26.312832434 +0000 UTC m=+330.991050219"
Jul 23 09:22:27.736856 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:27.736823    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:28.294381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.294318    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery" containerID="cri-o://46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88" gracePeriod=30
Jul 23 09:22:28.541549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.541523    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:28.727940 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.727900    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.727951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.727992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvr5s\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728332 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert\") pod \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\" (UID: \"3bac073d-efd8-4c16-8e39-3af677f3ad8e\") "
Jul 23 09:22:28.728332 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728280    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:22:28.728441 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.728372    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-ca-configmap\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.730296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs" (OuterVolumeSpecName: "local-certs") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:22:28.730408 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:22:28.730551 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730526    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts" (OuterVolumeSpecName: "cacerts") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:22:28.730634 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730592    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:22:28.730774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730759    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:22:28.730856 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.730820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s" (OuterVolumeSpecName: "kube-api-access-hvr5s") pod "3bac073d-efd8-4c16-8e39-3af677f3ad8e" (UID: "3bac073d-efd8-4c16-8e39-3af677f3ad8e"). InnerVolumeSpecName "kube-api-access-hvr5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:22:28.829739 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829715    2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-csr-dns-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.829739 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829740    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.830114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829755    2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bac073d-efd8-4c16-8e39-3af677f3ad8e-local-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.830114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829766    2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-cacerts\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.830114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829777    2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bac073d-efd8-4c16-8e39-3af677f3ad8e-istio-kubeconfig\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:28.830114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:28.829790    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr5s\" (UniqueName: \"kubernetes.io/projected/3bac073d-efd8-4c16-8e39-3af677f3ad8e-kube-api-access-hvr5s\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:22:29.240438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.240412    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-8vjr4"
Jul 23 09:22:29.300143 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.300113    2560 generic.go:356] "Generic (PLEG): container finished" podID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerID="46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88" exitCode=0
Jul 23 09:22:29.300277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.300147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" event={"ID":"3bac073d-efd8-4c16-8e39-3af677f3ad8e","Type":"ContainerDied","Data":"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"}
Jul 23 09:22:29.300277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.300157    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"
Jul 23 09:22:29.300277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.300180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-klfwz" event={"ID":"3bac073d-efd8-4c16-8e39-3af677f3ad8e","Type":"ContainerDied","Data":"c3915a548e6e576704b6e011747a90052312007013e587c5e35938ff344755b6"}
Jul 23 09:22:29.300277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.300199    2560 scope.go:117] "RemoveContainer" containerID="46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"
Jul 23 09:22:29.309128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.309112    2560 scope.go:117] "RemoveContainer" containerID="46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"
Jul 23 09:22:29.309330 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:22:29.309314    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88\": container with ID starting with 46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88 not found: ID does not exist" containerID="46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"
Jul 23 09:22:29.309373 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.309336    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88"} err="failed to get container status \"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88\": rpc error: code = NotFound desc = could not find container \"46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88\": container with ID starting with 46a0d1a980225e5aee31c9ba14f55e10c7405a299ca07f44d75c27c8538b6b88 not found: ID does not exist"
Jul 23 09:22:29.335496 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.335473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:29.342493 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.342474    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-klfwz"]
Jul 23 09:22:29.911533 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:29.911499    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" path="/var/lib/kubelet/pods/3bac073d-efd8-4c16-8e39-3af677f3ad8e/volumes"
Jul 23 09:22:37.292012 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:22:37.291980    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cb78dd96b-hrk74"
Jul 23 09:23:14.876882 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.876844    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:14.877336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.877173    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery"
Jul 23 09:23:14.877336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.877184    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery"
Jul 23 09:23:14.877336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.877251    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bac073d-efd8-4c16-8e39-3af677f3ad8e" containerName="discovery"
Jul 23 09:23:14.884679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.884654    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:14.887711 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.887685    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 09:23:14.888181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.888160    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9hdv7\""
Jul 23 09:23:14.888520 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.888502    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 09:23:14.891343 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.891319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:14.975205 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:14.975176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6f7\" (UniqueName: \"kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7\") pod \"kuadrant-operator-catalog-pq6zc\" (UID: \"be510674-a2e4-42e9-8233-bbf915115f7d\") " pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:15.077992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.077958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6f7\" (UniqueName: \"kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7\") pod \"kuadrant-operator-catalog-pq6zc\" (UID: \"be510674-a2e4-42e9-8233-bbf915115f7d\") " pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:15.086232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.086206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6f7\" (UniqueName: \"kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7\") pod \"kuadrant-operator-catalog-pq6zc\" (UID: \"be510674-a2e4-42e9-8233-bbf915115f7d\") " pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:15.196965 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.196929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:15.234770 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.234740    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:15.328439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.328415    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:15.330541 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:23:15.330505    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe510674_a2e4_42e9_8233_bbf915115f7d.slice/crio-8b7ae279251f3b72561a9aa9bc3f247d984457d595990952701550cfcf8b26dd WatchSource:0}: Error finding container 8b7ae279251f3b72561a9aa9bc3f247d984457d595990952701550cfcf8b26dd: Status 404 returned error can't find the container with id 8b7ae279251f3b72561a9aa9bc3f247d984457d595990952701550cfcf8b26dd
Jul 23 09:23:15.445413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.445387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lb4gb"]
Jul 23 09:23:15.448147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.448108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:15.458225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.458204    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lb4gb"]
Jul 23 09:23:15.489532 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.489504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" event={"ID":"be510674-a2e4-42e9-8233-bbf915115f7d","Type":"ContainerStarted","Data":"8b7ae279251f3b72561a9aa9bc3f247d984457d595990952701550cfcf8b26dd"}
Jul 23 09:23:15.581980 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.581954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bd7\" (UniqueName: \"kubernetes.io/projected/44ce8fa2-fd76-409b-8cca-5ffe1850f3bd-kube-api-access-l9bd7\") pod \"kuadrant-operator-catalog-lb4gb\" (UID: \"44ce8fa2-fd76-409b-8cca-5ffe1850f3bd\") " pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:15.683331 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.683305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bd7\" (UniqueName: \"kubernetes.io/projected/44ce8fa2-fd76-409b-8cca-5ffe1850f3bd-kube-api-access-l9bd7\") pod \"kuadrant-operator-catalog-lb4gb\" (UID: \"44ce8fa2-fd76-409b-8cca-5ffe1850f3bd\") " pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:15.691868 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.691845    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bd7\" (UniqueName: \"kubernetes.io/projected/44ce8fa2-fd76-409b-8cca-5ffe1850f3bd-kube-api-access-l9bd7\") pod \"kuadrant-operator-catalog-lb4gb\" (UID: \"44ce8fa2-fd76-409b-8cca-5ffe1850f3bd\") " pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:15.757481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.757413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:15.886858 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:15.886690    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lb4gb"]
Jul 23 09:23:15.889238 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:23:15.889215    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ce8fa2_fd76_409b_8cca_5ffe1850f3bd.slice/crio-ffe2c2bf17f211b1f5040a98928943cc82b19a8ecb3cb86b6e5526363574b027 WatchSource:0}: Error finding container ffe2c2bf17f211b1f5040a98928943cc82b19a8ecb3cb86b6e5526363574b027: Status 404 returned error can't find the container with id ffe2c2bf17f211b1f5040a98928943cc82b19a8ecb3cb86b6e5526363574b027
Jul 23 09:23:16.494881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:16.494850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb" event={"ID":"44ce8fa2-fd76-409b-8cca-5ffe1850f3bd","Type":"ContainerStarted","Data":"ffe2c2bf17f211b1f5040a98928943cc82b19a8ecb3cb86b6e5526363574b027"}
Jul 23 09:23:18.505440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.505406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" event={"ID":"be510674-a2e4-42e9-8233-bbf915115f7d","Type":"ContainerStarted","Data":"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"}
Jul 23 09:23:18.505930 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.505450    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" podUID="be510674-a2e4-42e9-8233-bbf915115f7d" containerName="registry-server" containerID="cri-o://45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4" gracePeriod=2
Jul 23 09:23:18.507071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.507036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb" event={"ID":"44ce8fa2-fd76-409b-8cca-5ffe1850f3bd","Type":"ContainerStarted","Data":"5ee986b6a64074327a70212bde0484b7e85efdb0ac94511d82a8502c9f293c17"}
Jul 23 09:23:18.540982 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.540934    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb" podStartSLOduration=1.349262404 podStartE2EDuration="3.540904869s" totalImagesPullingTime="2.191642465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:23:18.540738442 +0000 UTC m=+383.218956225" watchObservedRunningTime="2026-07-23 09:23:18.540904869 +0000 UTC m=+383.219122651"
Jul 23 09:23:18.542056 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.542027    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" podStartSLOduration=1.7884027420000002 podStartE2EDuration="4.542018912s" totalImagesPullingTime="2.75361617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:23:18.523627814 +0000 UTC m=+383.201845596" watchObservedRunningTime="2026-07-23 09:23:18.542018912 +0000 UTC m=+383.220236695"
Jul 23 09:23:18.752582 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.752557    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:18.812173 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.812106    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq6f7\" (UniqueName: \"kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7\") pod \"be510674-a2e4-42e9-8233-bbf915115f7d\" (UID: \"be510674-a2e4-42e9-8233-bbf915115f7d\") "
Jul 23 09:23:18.814188 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.814167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7" (OuterVolumeSpecName: "kube-api-access-rq6f7") pod "be510674-a2e4-42e9-8233-bbf915115f7d" (UID: "be510674-a2e4-42e9-8233-bbf915115f7d"). InnerVolumeSpecName "kube-api-access-rq6f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:23:18.913626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:18.913600    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq6f7\" (UniqueName: \"kubernetes.io/projected/be510674-a2e4-42e9-8233-bbf915115f7d-kube-api-access-rq6f7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:23:19.512622 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.512582    2560 generic.go:356] "Generic (PLEG): container finished" podID="be510674-a2e4-42e9-8233-bbf915115f7d" containerID="45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4" exitCode=0
Jul 23 09:23:19.513102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.512635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc"
Jul 23 09:23:19.513102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.512673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" event={"ID":"be510674-a2e4-42e9-8233-bbf915115f7d","Type":"ContainerDied","Data":"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"}
Jul 23 09:23:19.513102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.512746    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-pq6zc" event={"ID":"be510674-a2e4-42e9-8233-bbf915115f7d","Type":"ContainerDied","Data":"8b7ae279251f3b72561a9aa9bc3f247d984457d595990952701550cfcf8b26dd"}
Jul 23 09:23:19.513102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.512768    2560 scope.go:117] "RemoveContainer" containerID="45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"
Jul 23 09:23:19.521641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.521626    2560 scope.go:117] "RemoveContainer" containerID="45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"
Jul 23 09:23:19.521876 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:23:19.521861    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4\": container with ID starting with 45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4 not found: ID does not exist" containerID="45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"
Jul 23 09:23:19.521929 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.521885    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4"} err="failed to get container status \"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4\": rpc error: code = NotFound desc = could not find container \"45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4\": container with ID starting with 45ea629b01284ec29be1a60a72288d104ab73532e49b47638b51286fe3608db4 not found: ID does not exist"
Jul 23 09:23:19.532895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.532868    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:19.536295 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.536277    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-pq6zc"]
Jul 23 09:23:19.911663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:19.911631    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be510674-a2e4-42e9-8233-bbf915115f7d" path="/var/lib/kubelet/pods/be510674-a2e4-42e9-8233-bbf915115f7d/volumes"
Jul 23 09:23:25.758309 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:25.758268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:25.758844 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:25.758329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:25.781499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:25.781473    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:26.564144 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:26.564120    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-lb4gb"
Jul 23 09:23:45.964776 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.964740    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"]
Jul 23 09:23:45.965261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.965087    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be510674-a2e4-42e9-8233-bbf915115f7d" containerName="registry-server"
Jul 23 09:23:45.965261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.965098    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be510674-a2e4-42e9-8233-bbf915115f7d" containerName="registry-server"
Jul 23 09:23:45.965261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.965154    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be510674-a2e4-42e9-8233-bbf915115f7d" containerName="registry-server"
Jul 23 09:23:45.972213 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.972194    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:45.974969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.974948    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 09:23:45.975074 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.974980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-r9k6c\""
Jul 23 09:23:45.979085 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:45.978771    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"]
Jul 23 09:23:46.028285 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.028252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzh22\" (UniqueName: \"kubernetes.io/projected/1534f616-ae0c-4e0e-86a3-9dbed3d7e356-kube-api-access-zzh22\") pod \"dns-operator-controller-manager-864dd78c4c-7cgdk\" (UID: \"1534f616-ae0c-4e0e-86a3-9dbed3d7e356\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:46.128827 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.128784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzh22\" (UniqueName: \"kubernetes.io/projected/1534f616-ae0c-4e0e-86a3-9dbed3d7e356-kube-api-access-zzh22\") pod \"dns-operator-controller-manager-864dd78c4c-7cgdk\" (UID: \"1534f616-ae0c-4e0e-86a3-9dbed3d7e356\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:46.140441 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.140415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzh22\" (UniqueName: \"kubernetes.io/projected/1534f616-ae0c-4e0e-86a3-9dbed3d7e356-kube-api-access-zzh22\") pod \"dns-operator-controller-manager-864dd78c4c-7cgdk\" (UID: \"1534f616-ae0c-4e0e-86a3-9dbed3d7e356\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:46.283016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.282896    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:46.422399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.422349    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"]
Jul 23 09:23:46.424807 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:23:46.424779    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1534f616_ae0c_4e0e_86a3_9dbed3d7e356.slice/crio-e29cb830117672af0a1a663de63c18134b1358512c6177aead4dddcfbec8559f WatchSource:0}: Error finding container e29cb830117672af0a1a663de63c18134b1358512c6177aead4dddcfbec8559f: Status 404 returned error can't find the container with id e29cb830117672af0a1a663de63c18134b1358512c6177aead4dddcfbec8559f
Jul 23 09:23:46.629521 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:46.629439    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk" event={"ID":"1534f616-ae0c-4e0e-86a3-9dbed3d7e356","Type":"ContainerStarted","Data":"e29cb830117672af0a1a663de63c18134b1358512c6177aead4dddcfbec8559f"}
Jul 23 09:23:48.642587 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:48.642559    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk" event={"ID":"1534f616-ae0c-4e0e-86a3-9dbed3d7e356","Type":"ContainerStarted","Data":"f303f139b82a60a1af71d71700eaf237ed8d6a77787782a32d339bde7f752ac5"}
Jul 23 09:23:48.642890 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:48.642622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:23:48.662329 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:48.662286    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk" podStartSLOduration=1.5131896299999998 podStartE2EDuration="3.662272175s" totalImagesPullingTime="2.149082545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:23:48.661409033 +0000 UTC m=+413.339626845" watchObservedRunningTime="2026-07-23 09:23:48.662272175 +0000 UTC m=+413.340490027"
Jul 23 09:23:49.562240 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.562209    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"]
Jul 23 09:23:49.565498 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.565481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:49.575607 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.575588    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-njbsr\""
Jul 23 09:23:49.582396 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.582370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"]
Jul 23 09:23:49.661946 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.661903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhwc\" (UniqueName: \"kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc\") pod \"limitador-operator-controller-manager-859d7b78bd-65n4d\" (UID: \"9b5e8125-be2e-4bd4-b292-59ae5f79a37d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:49.763166 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.763128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhwc\" (UniqueName: \"kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc\") pod \"limitador-operator-controller-manager-859d7b78bd-65n4d\" (UID: \"9b5e8125-be2e-4bd4-b292-59ae5f79a37d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:49.778774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.778751    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhwc\" (UniqueName: \"kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc\") pod \"limitador-operator-controller-manager-859d7b78bd-65n4d\" (UID: \"9b5e8125-be2e-4bd4-b292-59ae5f79a37d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:49.875080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:49.874997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:50.009386 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:50.009359    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"]
Jul 23 09:23:50.654068 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:50.654029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d" event={"ID":"9b5e8125-be2e-4bd4-b292-59ae5f79a37d","Type":"ContainerStarted","Data":"307a2e8d9f6e60325987aaadd3c8973e7ffe00874f265fccc2c77a4249b22b63"}
Jul 23 09:23:52.665326 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:52.665291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d" event={"ID":"9b5e8125-be2e-4bd4-b292-59ae5f79a37d","Type":"ContainerStarted","Data":"3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d"}
Jul 23 09:23:52.665732 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:52.665362    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:23:52.690305 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:52.690259    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d" podStartSLOduration=1.98132094 podStartE2EDuration="3.690244981s" totalImagesPullingTime="1.708924041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:23:52.689103044 +0000 UTC m=+417.367320861" watchObservedRunningTime="2026-07-23 09:23:52.690244981 +0000 UTC m=+417.368462763"
Jul 23 09:23:54.368593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.368558    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"]
Jul 23 09:23:54.373877 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.373856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.376181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.376161    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8kkf8\""
Jul 23 09:23:54.386956 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.386932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"]
Jul 23 09:23:54.402895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.402872    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.403007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.402902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t42\" (UniqueName: \"kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.503834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.503804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.503960 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.503839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t42\" (UniqueName: \"kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.504153 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.504135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.514884 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.514859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t42\" (UniqueName: \"kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.683427 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.683406    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:23:54.817962 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:54.817935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"]
Jul 23 09:23:54.821339 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:23:54.821312    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283a46a9_449c_461b_b78b_b2f488be24f1.slice/crio-3fe5b5e66f6fb07b5d3037c71bb49e04956798d4a61870946966f7e6f9e48948 WatchSource:0}: Error finding container 3fe5b5e66f6fb07b5d3037c71bb49e04956798d4a61870946966f7e6f9e48948: Status 404 returned error can't find the container with id 3fe5b5e66f6fb07b5d3037c71bb49e04956798d4a61870946966f7e6f9e48948
Jul 23 09:23:55.678132 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:55.678100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx" event={"ID":"283a46a9-449c-461b-b78b-b2f488be24f1","Type":"ContainerStarted","Data":"3fe5b5e66f6fb07b5d3037c71bb49e04956798d4a61870946966f7e6f9e48948"}
Jul 23 09:23:59.649741 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:23:59.649703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7cgdk"
Jul 23 09:24:00.702393 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:00.702358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx" event={"ID":"283a46a9-449c-461b-b78b-b2f488be24f1","Type":"ContainerStarted","Data":"f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3"}
Jul 23 09:24:00.702833 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:00.702489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:00.724684 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:00.724638    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx" podStartSLOduration=1.377074128 podStartE2EDuration="6.72462108s" totalImagesPullingTime="5.347546952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:00.72411399 +0000 UTC m=+425.402331775" watchObservedRunningTime="2026-07-23 09:24:00.72462108 +0000 UTC m=+425.402838862"
Jul 23 09:24:02.976239 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.976205    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"]
Jul 23 09:24:02.979662 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.979644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:02.982143 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.982121    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 09:24:02.982143 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.982134    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 09:24:02.982300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.982154    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g68sf\""
Jul 23 09:24:02.989259 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:02.989236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"]
Jul 23 09:24:03.075134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.075109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thgsh\" (UniqueName: \"kubernetes.io/projected/91e4a155-59b8-4e44-84c1-007a13e33cd5-kube-api-access-thgsh\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.075134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.075148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91e4a155-59b8-4e44-84c1-007a13e33cd5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.075348 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.075175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e4a155-59b8-4e44-84c1-007a13e33cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.176549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.176520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thgsh\" (UniqueName: \"kubernetes.io/projected/91e4a155-59b8-4e44-84c1-007a13e33cd5-kube-api-access-thgsh\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.176680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.176555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91e4a155-59b8-4e44-84c1-007a13e33cd5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.176680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.176580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e4a155-59b8-4e44-84c1-007a13e33cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.177181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.177152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/91e4a155-59b8-4e44-84c1-007a13e33cd5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.178735 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.178716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/91e4a155-59b8-4e44-84c1-007a13e33cd5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.186349 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.186329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thgsh\" (UniqueName: \"kubernetes.io/projected/91e4a155-59b8-4e44-84c1-007a13e33cd5-kube-api-access-thgsh\") pod \"kuadrant-console-plugin-66b5774985-p5hbq\" (UID: \"91e4a155-59b8-4e44-84c1-007a13e33cd5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.289270 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.289209    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"
Jul 23 09:24:03.420578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.420556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq"]
Jul 23 09:24:03.421762 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:24:03.421727    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e4a155_59b8_4e44_84c1_007a13e33cd5.slice/crio-4d04dca2ef87f259f640f5b8159d4068858198d063ddc2b518d296f3dac7399e WatchSource:0}: Error finding container 4d04dca2ef87f259f640f5b8159d4068858198d063ddc2b518d296f3dac7399e: Status 404 returned error can't find the container with id 4d04dca2ef87f259f640f5b8159d4068858198d063ddc2b518d296f3dac7399e
Jul 23 09:24:03.671815 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.671791    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:03.716489 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:03.716464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq" event={"ID":"91e4a155-59b8-4e44-84c1-007a13e33cd5","Type":"ContainerStarted","Data":"4d04dca2ef87f259f640f5b8159d4068858198d063ddc2b518d296f3dac7399e"}
Jul 23 09:24:11.708934 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:11.708877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:12.761483 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.761445    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"]
Jul 23 09:24:12.788573 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.788540    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.789017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.788989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"]
Jul 23 09:24:12.862753 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.862716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.862753 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.862763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ks9\" (UniqueName: \"kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.963951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.963896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.964139 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.963972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ks9\" (UniqueName: \"kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.964321 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.964297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:12.998968 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:12.998938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ks9\" (UniqueName: \"kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:13.102355 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.102274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:13.418629 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.418597    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"]
Jul 23 09:24:13.434759 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.434727    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"]
Jul 23 09:24:13.457992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.457956    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"]
Jul 23 09:24:13.458208 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.458158    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" containerName="manager" containerID="cri-o://f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3" gracePeriod=2
Jul 23 09:24:13.469738 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.469698    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"]
Jul 23 09:24:13.477961 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.477933    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:13.478489 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.478466    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" containerName="manager"
Jul 23 09:24:13.478564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.478492    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" containerName="manager"
Jul 23 09:24:13.478626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.478617    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" containerName="manager"
Jul 23 09:24:13.486078 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.486035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.487475 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.487436    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"]
Jul 23 09:24:13.488784 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.487622    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerName="manager" containerID="cri-o://3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d" gracePeriod=2
Jul 23 09:24:13.504431 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.503198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:13.505755 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.505732    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"]
Jul 23 09:24:13.514654 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.514633    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:13.515008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.514994    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerName="manager"
Jul 23 09:24:13.515008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.515009    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerName="manager"
Jul 23 09:24:13.515128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.515074    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerName="manager"
Jul 23 09:24:13.519626 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.519608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.541214 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:13.541182    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:13.544626 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:13.543540    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:13.552956 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.552896    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"]
Jul 23 09:24:13.558707 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.558651    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:13.558820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.558746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:13.568283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.568154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.568283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.568189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t564f\" (UniqueName: \"kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.574385 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.574326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"]
Jul 23 09:24:13.596903 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:13.596866    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:13.600794 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:13.600771    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:13.606700 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:13.606660    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:13.668888 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.668807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szz8p\" (UniqueName: \"kubernetes.io/projected/985c8d9d-ff72-4873-a70d-e29994d39ed4-kube-api-access-szz8p\") pod \"limitador-operator-controller-manager-859d7b78bd-lhn2f\" (UID: \"985c8d9d-ff72-4873-a70d-e29994d39ed4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:13.668888 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.668867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc89z\" (UniqueName: \"kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.669111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.668958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.669111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.668985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t564f\" (UniqueName: \"kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.669111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.669047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.669280 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.669261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.690062 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.690034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t564f\" (UniqueName: \"kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6crx7\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.769588 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.769559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.770055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.769615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szz8p\" (UniqueName: \"kubernetes.io/projected/985c8d9d-ff72-4873-a70d-e29994d39ed4-kube-api-access-szz8p\") pod \"limitador-operator-controller-manager-859d7b78bd-lhn2f\" (UID: \"985c8d9d-ff72-4873-a70d-e29994d39ed4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:13.770055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.769652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc89z\" (UniqueName: \"kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.770055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.770042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.790545 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.790515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szz8p\" (UniqueName: \"kubernetes.io/projected/985c8d9d-ff72-4873-a70d-e29994d39ed4-kube-api-access-szz8p\") pod \"limitador-operator-controller-manager-859d7b78bd-lhn2f\" (UID: \"985c8d9d-ff72-4873-a70d-e29994d39ed4\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:13.793134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.793108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc89z\" (UniqueName: \"kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bq6rk\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.882849 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.882819    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:13.893977 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.893946    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:13.900887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:13.900861    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:14.669691 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:14.669646    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 09:24:24.651131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.651099    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.654454 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.654423    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.661101 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.661076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:24.684443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.684413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:24.779394 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.779324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume\") pod \"283a46a9-449c-461b-b78b-b2f488be24f1\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") "
Jul 23 09:24:24.779394 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.779378    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4t42\" (UniqueName: \"kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42\") pod \"283a46a9-449c-461b-b78b-b2f488be24f1\" (UID: \"283a46a9-449c-461b-b78b-b2f488be24f1\") "
Jul 23 09:24:24.779830 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.779802    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "283a46a9-449c-461b-b78b-b2f488be24f1" (UID: "283a46a9-449c-461b-b78b-b2f488be24f1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:24:24.781373 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.781351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42" (OuterVolumeSpecName: "kube-api-access-p4t42") pod "283a46a9-449c-461b-b78b-b2f488be24f1" (UID: "283a46a9-449c-461b-b78b-b2f488be24f1"). InnerVolumeSpecName "kube-api-access-p4t42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:24:24.807332 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:24:24.807296    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b99a740_c568_4d95_8a8c_2af422ef38ea.slice/crio-7453e11c4e290821d1347c5b8736dc1f820e21a6f68f77e591f0726b55d41fc0 WatchSource:0}: Error finding container 7453e11c4e290821d1347c5b8736dc1f820e21a6f68f77e591f0726b55d41fc0: Status 404 returned error can't find the container with id 7453e11c4e290821d1347c5b8736dc1f820e21a6f68f77e591f0726b55d41fc0
Jul 23 09:24:24.807950 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:24:24.807909    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aea2033_28cc_4f0e_ae19_44405a2395b0.slice/crio-8acb5d81b2229643f7035876995cc05dab670319292dae3d679d4a7bb929175e WatchSource:0}: Error finding container 8acb5d81b2229643f7035876995cc05dab670319292dae3d679d4a7bb929175e: Status 404 returned error can't find the container with id 8acb5d81b2229643f7035876995cc05dab670319292dae3d679d4a7bb929175e
Jul 23 09:24:24.831547 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.831519    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.833888 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.833861    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.836609 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.836562    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.837352 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.837328    2560 generic.go:356] "Generic (PLEG): container finished" podID="283a46a9-449c-461b-b78b-b2f488be24f1" containerID="f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3" exitCode=0
Jul 23 09:24:24.837446 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.837400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.837446 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.837418    2560 scope.go:117] "RemoveContainer" containerID="f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3"
Jul 23 09:24:24.838517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.838494    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"]
Jul 23 09:24:24.839036 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.839009    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" event={"ID":"2b99a740-c568-4d95-8a8c-2af422ef38ea","Type":"ContainerStarted","Data":"7453e11c4e290821d1347c5b8736dc1f820e21a6f68f77e591f0726b55d41fc0"}
Jul 23 09:24:24.840360 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.840333    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.841011 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.840987    2560 generic.go:356] "Generic (PLEG): container finished" podID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" containerID="3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d" exitCode=0
Jul 23 09:24:24.841243 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.841227    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.843047 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:24:24.842669    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985c8d9d_ff72_4873_a70d_e29994d39ed4.slice/crio-5f27406e8f3a18245c212d83de356fb8ef0ddb370ea0e640d769688915952a77 WatchSource:0}: Error finding container 5f27406e8f3a18245c212d83de356fb8ef0ddb370ea0e640d769688915952a77: Status 404 returned error can't find the container with id 5f27406e8f3a18245c212d83de356fb8ef0ddb370ea0e640d769688915952a77
Jul 23 09:24:24.843047 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.842716    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.844899 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.844861    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.845023 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.844899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" event={"ID":"5aea2033-28cc-4f0e-ae19-44405a2395b0","Type":"ContainerStarted","Data":"8acb5d81b2229643f7035876995cc05dab670319292dae3d679d4a7bb929175e"}
Jul 23 09:24:24.847064 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.847041    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.848518 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.848499    2560 scope.go:117] "RemoveContainer" containerID="f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3"
Jul 23 09:24:24.848978 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.848893    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3\": container with ID starting with f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3 not found: ID does not exist" containerID="f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3"
Jul 23 09:24:24.848978 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.848939    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3"} err="failed to get container status \"f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3\": rpc error: code = NotFound desc = could not find container \"f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3\": container with ID starting with f79414e90a793d91a333f00b1f74d1b0840aab0f64d3e9985970fe4e2238f7d3 not found: ID does not exist"
Jul 23 09:24:24.848978 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.848962    2560 scope.go:117] "RemoveContainer" containerID="3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d"
Jul 23 09:24:24.851280 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.851256    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:24.853319 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.853290    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:24.856724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.856705    2560 scope.go:117] "RemoveContainer" containerID="3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d"
Jul 23 09:24:24.857014 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:24.856993    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d\": container with ID starting with 3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d not found: ID does not exist" containerID="3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d"
Jul 23 09:24:24.857081 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.857018    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d"} err="failed to get container status \"3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d\": rpc error: code = NotFound desc = could not find container \"3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d\": container with ID starting with 3634807cdc0b0bfd416e01e621bd44a07e74a4327b824c7bcb72af35874e372d not found: ID does not exist"
Jul 23 09:24:24.879871 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.879854    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/283a46a9-449c-461b-b78b-b2f488be24f1-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:24.879936 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.879874    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4t42\" (UniqueName: \"kubernetes.io/projected/283a46a9-449c-461b-b78b-b2f488be24f1-kube-api-access-p4t42\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:24.981196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.981172    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rhwc\" (UniqueName: \"kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc\") pod \"9b5e8125-be2e-4bd4-b292-59ae5f79a37d\" (UID: \"9b5e8125-be2e-4bd4-b292-59ae5f79a37d\") "
Jul 23 09:24:24.983095 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:24.983070    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc" (OuterVolumeSpecName: "kube-api-access-9rhwc") pod "9b5e8125-be2e-4bd4-b292-59ae5f79a37d" (UID: "9b5e8125-be2e-4bd4-b292-59ae5f79a37d"). InnerVolumeSpecName "kube-api-access-9rhwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:24:25.082121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.082097    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rhwc\" (UniqueName: \"kubernetes.io/projected/9b5e8125-be2e-4bd4-b292-59ae5f79a37d-kube-api-access-9rhwc\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:25.153257 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.153225    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:25.155194 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.155168    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:25.852413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.852326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" event={"ID":"2b99a740-c568-4d95-8a8c-2af422ef38ea","Type":"ContainerStarted","Data":"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"}
Jul 23 09:24:25.852823 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.852456    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:25.853960 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.853938    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f" event={"ID":"985c8d9d-ff72-4873-a70d-e29994d39ed4","Type":"ContainerStarted","Data":"21d5d84138ec922a7f91ef39515117c45c88250348f26500b9fbae8de14f41af"}
Jul 23 09:24:25.854080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.854012    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:25.854080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.854030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f" event={"ID":"985c8d9d-ff72-4873-a70d-e29994d39ed4","Type":"ContainerStarted","Data":"5f27406e8f3a18245c212d83de356fb8ef0ddb370ea0e640d769688915952a77"}
Jul 23 09:24:25.855628 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.855593    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:25.856764 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.856740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" event={"ID":"5aea2033-28cc-4f0e-ae19-44405a2395b0","Type":"ContainerStarted","Data":"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"}
Jul 23 09:24:25.856878 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.856865    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:25.857660 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.857625    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:25.858811 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.858789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq" event={"ID":"91e4a155-59b8-4e44-84c1-007a13e33cd5","Type":"ContainerStarted","Data":"f1babd88f58fbddf486de8e4007f7b1d410dbb4da44ddf04814b9f8d7145df2e"}
Jul 23 09:24:25.882639 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.882596    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" podStartSLOduration=12.882583324 podStartE2EDuration="12.882583324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:25.881898923 +0000 UTC m=+450.560116727" watchObservedRunningTime="2026-07-23 09:24:25.882583324 +0000 UTC m=+450.560801104"
Jul 23 09:24:25.884939 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.884898    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-65n4d\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-65n4d"
Jul 23 09:24:25.903365 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.903316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-p5hbq" podStartSLOduration=2.450365562 podStartE2EDuration="23.903302712s" totalImagesPullingTime="21.45293715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:25.902508371 +0000 UTC m=+450.580726154" watchObservedRunningTime="2026-07-23 09:24:25.903302712 +0000 UTC m=+450.581520494"
Jul 23 09:24:25.912686 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.912665    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" path="/var/lib/kubelet/pods/283a46a9-449c-461b-b78b-b2f488be24f1/volumes"
Jul 23 09:24:25.913047 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.913033    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5e8125-be2e-4bd4-b292-59ae5f79a37d" path="/var/lib/kubelet/pods/9b5e8125-be2e-4bd4-b292-59ae5f79a37d/volumes"
Jul 23 09:24:25.921525 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.921476    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f" podStartSLOduration=12.921458743 podStartE2EDuration="12.921458743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:25.919304858 +0000 UTC m=+450.597522641" watchObservedRunningTime="2026-07-23 09:24:25.921458743 +0000 UTC m=+450.599676526"
Jul 23 09:24:25.938948 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:25.938371    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" podStartSLOduration=12.93835237 podStartE2EDuration="12.93835237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:25.936333307 +0000 UTC m=+450.614551090" watchObservedRunningTime="2026-07-23 09:24:25.93835237 +0000 UTC m=+450.616570157"
Jul 23 09:24:25.938948 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:25.938771    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wrgpx\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="283a46a9-449c-461b-b78b-b2f488be24f1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wrgpx"
Jul 23 09:24:26.996138 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:26.996102    2560 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-ftfdv_kuadrant-system_7fc29716-31f6-4c83-8488-5ce8f0daf711_0(7e38d0f890c7f57592201b01edbbae6570294be83199c7654f68df3546f04bb4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-ftfdv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 23 09:24:26.996409 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:26.996164    2560 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-ftfdv_kuadrant-system_7fc29716-31f6-4c83-8488-5ce8f0daf711_0(7e38d0f890c7f57592201b01edbbae6570294be83199c7654f68df3546f04bb4): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-ftfdv to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:27.867731 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:27.867700    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:27.870159 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:27.870125    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="7fc29716-31f6-4c83-8488-5ce8f0daf711" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:27.874141 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:27.874122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:27.876373 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:27.876353    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="7fc29716-31f6-4c83-8488-5ce8f0daf711" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:28.008556 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.008528    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume\") pod \"7fc29716-31f6-4c83-8488-5ce8f0daf711\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") "
Jul 23 09:24:28.008886 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.008577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65ks9\" (UniqueName: \"kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9\") pod \"7fc29716-31f6-4c83-8488-5ce8f0daf711\" (UID: \"7fc29716-31f6-4c83-8488-5ce8f0daf711\") "
Jul 23 09:24:28.008886 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.008763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7fc29716-31f6-4c83-8488-5ce8f0daf711" (UID: "7fc29716-31f6-4c83-8488-5ce8f0daf711"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:24:28.010553 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.010533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9" (OuterVolumeSpecName: "kube-api-access-65ks9") pod "7fc29716-31f6-4c83-8488-5ce8f0daf711" (UID: "7fc29716-31f6-4c83-8488-5ce8f0daf711"). InnerVolumeSpecName "kube-api-access-65ks9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:24:28.109297 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.109268    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fc29716-31f6-4c83-8488-5ce8f0daf711-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:28.109297 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.109294    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65ks9\" (UniqueName: \"kubernetes.io/projected/7fc29716-31f6-4c83-8488-5ce8f0daf711-kube-api-access-65ks9\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:28.872157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:28.872121    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:28.874706 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:28.874662    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="7fc29716-31f6-4c83-8488-5ce8f0daf711" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:28.880052 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:28.880030    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-ftfdv\" is forbidden: User \"system:node:ip-10-0-129-47.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-47.ec2.internal' and this object" podUID="7fc29716-31f6-4c83-8488-5ce8f0daf711" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-ftfdv"
Jul 23 09:24:29.912127 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:29.912095    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc29716-31f6-4c83-8488-5ce8f0daf711" path="/var/lib/kubelet/pods/7fc29716-31f6-4c83-8488-5ce8f0daf711/volumes"
Jul 23 09:24:36.866245 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:36.866205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:36.866797 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:36.866275    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lhn2f"
Jul 23 09:24:36.866797 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:36.866707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:36.958590 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:36.958551    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:36.958765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:36.958740    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" podUID="2b99a740-c568-4d95-8a8c-2af422ef38ea" containerName="manager" containerID="cri-o://ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa" gracePeriod=10
Jul 23 09:24:37.221999 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.221977    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:37.274888 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.274860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume\") pod \"2b99a740-c568-4d95-8a8c-2af422ef38ea\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") "
Jul 23 09:24:37.275073 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.274928    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc89z\" (UniqueName: \"kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z\") pod \"2b99a740-c568-4d95-8a8c-2af422ef38ea\" (UID: \"2b99a740-c568-4d95-8a8c-2af422ef38ea\") "
Jul 23 09:24:37.275327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.275290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2b99a740-c568-4d95-8a8c-2af422ef38ea" (UID: "2b99a740-c568-4d95-8a8c-2af422ef38ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:24:37.277160 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.277133    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z" (OuterVolumeSpecName: "kube-api-access-jc89z") pod "2b99a740-c568-4d95-8a8c-2af422ef38ea" (UID: "2b99a740-c568-4d95-8a8c-2af422ef38ea"). InnerVolumeSpecName "kube-api-access-jc89z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:24:37.278054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.278032    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:24:37.278403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.278390    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b99a740-c568-4d95-8a8c-2af422ef38ea" containerName="manager"
Jul 23 09:24:37.278403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.278404    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b99a740-c568-4d95-8a8c-2af422ef38ea" containerName="manager"
Jul 23 09:24:37.278522 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.278461    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b99a740-c568-4d95-8a8c-2af422ef38ea" containerName="manager"
Jul 23 09:24:37.280364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.280347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.294697 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.294669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:24:37.375572 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.375513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d85n\" (UniqueName: \"kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.375572 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.375566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.375692 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.375638    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b99a740-c568-4d95-8a8c-2af422ef38ea-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:37.375692 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.375653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc89z\" (UniqueName: \"kubernetes.io/projected/2b99a740-c568-4d95-8a8c-2af422ef38ea-kube-api-access-jc89z\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:37.476216 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.476190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.476325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.476260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d85n\" (UniqueName: \"kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.476598 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.476579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.489595 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.489569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d85n\" (UniqueName: \"kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n\") pod \"kuadrant-operator-controller-manager-86878c769f-smxbg\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.593722 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.593697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.725767 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.725744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:24:37.728101 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:24:37.728078    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734baa38_779b_4815_9050_bf348901823e.slice/crio-798ce4c92dc08713787db91754c51ee71bd66c329bd585cdc407532e7d304636 WatchSource:0}: Error finding container 798ce4c92dc08713787db91754c51ee71bd66c329bd585cdc407532e7d304636: Status 404 returned error can't find the container with id 798ce4c92dc08713787db91754c51ee71bd66c329bd585cdc407532e7d304636
Jul 23 09:24:37.914118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.914083    2560 generic.go:356] "Generic (PLEG): container finished" podID="2b99a740-c568-4d95-8a8c-2af422ef38ea" containerID="ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa" exitCode=0
Jul 23 09:24:37.914544 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.914124    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"
Jul 23 09:24:37.914544 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.914165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" event={"ID":"2b99a740-c568-4d95-8a8c-2af422ef38ea","Type":"ContainerDied","Data":"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"}
Jul 23 09:24:37.914544 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.914213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk" event={"ID":"2b99a740-c568-4d95-8a8c-2af422ef38ea","Type":"ContainerDied","Data":"7453e11c4e290821d1347c5b8736dc1f820e21a6f68f77e591f0726b55d41fc0"}
Jul 23 09:24:37.914544 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.914242    2560 scope.go:117] "RemoveContainer" containerID="ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"
Jul 23 09:24:37.915840 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.915818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" event={"ID":"734baa38-779b-4815-9050-bf348901823e","Type":"ContainerStarted","Data":"611867224baebb251f9b1bce43120dde498fab67bfcb845ba2ca7eed63365dce"}
Jul 23 09:24:37.916007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.915857    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:37.916007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.915873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" event={"ID":"734baa38-779b-4815-9050-bf348901823e","Type":"ContainerStarted","Data":"798ce4c92dc08713787db91754c51ee71bd66c329bd585cdc407532e7d304636"}
Jul 23 09:24:37.930018 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.929993    2560 scope.go:117] "RemoveContainer" containerID="ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"
Jul 23 09:24:37.930301 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:37.930282    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa\": container with ID starting with ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa not found: ID does not exist" containerID="ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"
Jul 23 09:24:37.930372 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.930306    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa"} err="failed to get container status \"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa\": rpc error: code = NotFound desc = could not find container \"ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa\": container with ID starting with ab4cbb8495e64740f2033ec27ce157c2b642342b5e16ebd2ea08af7037dc51fa not found: ID does not exist"
Jul 23 09:24:37.941645 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.941599    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" podStartSLOduration=0.941587517 podStartE2EDuration="941.587517ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:24:37.941442048 +0000 UTC m=+462.619659832" watchObservedRunningTime="2026-07-23 09:24:37.941587517 +0000 UTC m=+462.619805300"
Jul 23 09:24:37.957669 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.957643    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:37.963997 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:37.963978    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bq6rk"]
Jul 23 09:24:39.911928 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:39.911884    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b99a740-c568-4d95-8a8c-2af422ef38ea" path="/var/lib/kubelet/pods/2b99a740-c568-4d95-8a8c-2af422ef38ea/volumes"
Jul 23 09:24:48.924300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:48.924272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:24:49.020823 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.020789    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:49.021092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.021066    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" podUID="5aea2033-28cc-4f0e-ae19-44405a2395b0" containerName="manager" containerID="cri-o://bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165" gracePeriod=10
Jul 23 09:24:49.863130 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.863100    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:49.971374 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.971304    2560 generic.go:356] "Generic (PLEG): container finished" podID="5aea2033-28cc-4f0e-ae19-44405a2395b0" containerID="bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165" exitCode=0
Jul 23 09:24:49.971374 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.971360    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"
Jul 23 09:24:49.971870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.971385    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" event={"ID":"5aea2033-28cc-4f0e-ae19-44405a2395b0","Type":"ContainerDied","Data":"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"}
Jul 23 09:24:49.971870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.971434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7" event={"ID":"5aea2033-28cc-4f0e-ae19-44405a2395b0","Type":"ContainerDied","Data":"8acb5d81b2229643f7035876995cc05dab670319292dae3d679d4a7bb929175e"}
Jul 23 09:24:49.971870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.971455    2560 scope.go:117] "RemoveContainer" containerID="bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"
Jul 23 09:24:49.974689 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.974662    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume\") pod \"5aea2033-28cc-4f0e-ae19-44405a2395b0\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") "
Jul 23 09:24:49.974809 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.974722    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t564f\" (UniqueName: \"kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f\") pod \"5aea2033-28cc-4f0e-ae19-44405a2395b0\" (UID: \"5aea2033-28cc-4f0e-ae19-44405a2395b0\") "
Jul 23 09:24:49.975022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.975000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5aea2033-28cc-4f0e-ae19-44405a2395b0" (UID: "5aea2033-28cc-4f0e-ae19-44405a2395b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:24:49.977058 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.977029    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f" (OuterVolumeSpecName: "kube-api-access-t564f") pod "5aea2033-28cc-4f0e-ae19-44405a2395b0" (UID: "5aea2033-28cc-4f0e-ae19-44405a2395b0"). InnerVolumeSpecName "kube-api-access-t564f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:24:49.980422 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.980404    2560 scope.go:117] "RemoveContainer" containerID="bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"
Jul 23 09:24:49.980643 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:24:49.980624    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165\": container with ID starting with bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165 not found: ID does not exist" containerID="bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"
Jul 23 09:24:49.980702 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:49.980659    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165"} err="failed to get container status \"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165\": rpc error: code = NotFound desc = could not find container \"bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165\": container with ID starting with bbc4ebe521269152c96c361cbd1d74afc4012f4b6925bb6560e04ce51856b165 not found: ID does not exist"
Jul 23 09:24:50.075511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:50.075487    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5aea2033-28cc-4f0e-ae19-44405a2395b0-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:50.075511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:50.075508    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t564f\" (UniqueName: \"kubernetes.io/projected/5aea2033-28cc-4f0e-ae19-44405a2395b0-kube-api-access-t564f\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:24:50.294303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:50.294255    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:50.298179 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:50.298154    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6crx7"]
Jul 23 09:24:51.912326 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:24:51.912294    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aea2033-28cc-4f0e-ae19-44405a2395b0" path="/var/lib/kubelet/pods/5aea2033-28cc-4f0e-ae19-44405a2395b0/volumes"
Jul 23 09:25:04.690271 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.690235    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:04.690819 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.690576    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aea2033-28cc-4f0e-ae19-44405a2395b0" containerName="manager"
Jul 23 09:25:04.690819 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.690587    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aea2033-28cc-4f0e-ae19-44405a2395b0" containerName="manager"
Jul 23 09:25:04.690819 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.690663    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aea2033-28cc-4f0e-ae19-44405a2395b0" containerName="manager"
Jul 23 09:25:04.746302 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.746268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:04.746424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.746358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.749460 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.749432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 09:25:04.785017 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.784975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.785227 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.785085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6lk\" (UniqueName: \"kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.788834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.788810    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:04.886447 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.886418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.886558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.886477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6lk\" (UniqueName: \"kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.887056 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.887038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:04.894438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:04.894419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6lk\" (UniqueName: \"kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk\") pod \"limitador-limitador-64c94fd64f-tct4p\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:05.056879 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:05.056826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:05.188274 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:05.188250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:05.190446 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:25:05.190417    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a46d417_45f9_421d_9e43_ab64ccca5c0d.slice/crio-a9b0a07ea50c74e49644d6b2a4b9a3ad20c4a6462630a2c5d519a5c763f6be44 WatchSource:0}: Error finding container a9b0a07ea50c74e49644d6b2a4b9a3ad20c4a6462630a2c5d519a5c763f6be44: Status 404 returned error can't find the container with id a9b0a07ea50c74e49644d6b2a4b9a3ad20c4a6462630a2c5d519a5c763f6be44
Jul 23 09:25:06.052357 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:06.052320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" event={"ID":"4a46d417-45f9-421d-9e43-ab64ccca5c0d","Type":"ContainerStarted","Data":"a9b0a07ea50c74e49644d6b2a4b9a3ad20c4a6462630a2c5d519a5c763f6be44"}
Jul 23 09:25:07.925529 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:07.925486    2560 pod_container_manager_linux.go:217] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod2b99a740-c568-4d95-8a8c-2af422ef38ea"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod2b99a740-c568-4d95-8a8c-2af422ef38ea] : Timed out while waiting for systemd to remove kubepods-burstable-pod2b99a740_c568_4d95_8a8c_2af422ef38ea.slice"
Jul 23 09:25:09.069438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:09.069399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" event={"ID":"4a46d417-45f9-421d-9e43-ab64ccca5c0d","Type":"ContainerStarted","Data":"7a333feaed39b4ed0cb3609fbb2ab0a54affcee4fd5a44b4c27bde15ae6e359b"}
Jul 23 09:25:09.069807 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:09.069468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:09.088346 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:09.088299    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" podStartSLOduration=2.268673403 podStartE2EDuration="5.088285092s" totalImagesPullingTime="2.819611689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:09.086951064 +0000 UTC m=+493.765168866" watchObservedRunningTime="2026-07-23 09:25:09.088285092 +0000 UTC m=+493.766502892"
Jul 23 09:25:20.074795 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:20.074762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:22.637144 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:22.637111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:22.637610 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:22.637285    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" podUID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" containerName="limitador" containerID="cri-o://7a333feaed39b4ed0cb3609fbb2ab0a54affcee4fd5a44b4c27bde15ae6e359b" gracePeriod=30
Jul 23 09:25:23.134804 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.134775    2560 generic.go:356] "Generic (PLEG): container finished" podID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" containerID="7a333feaed39b4ed0cb3609fbb2ab0a54affcee4fd5a44b4c27bde15ae6e359b" exitCode=0
Jul 23 09:25:23.134964 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.134849    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" event={"ID":"4a46d417-45f9-421d-9e43-ab64ccca5c0d","Type":"ContainerDied","Data":"7a333feaed39b4ed0cb3609fbb2ab0a54affcee4fd5a44b4c27bde15ae6e359b"}
Jul 23 09:25:23.174079 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.174060    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:23.325211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.325142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file\") pod \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") "
Jul 23 09:25:23.325211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.325185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd6lk\" (UniqueName: \"kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk\") pod \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\" (UID: \"4a46d417-45f9-421d-9e43-ab64ccca5c0d\") "
Jul 23 09:25:23.325523 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.325500    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file" (OuterVolumeSpecName: "config-file") pod "4a46d417-45f9-421d-9e43-ab64ccca5c0d" (UID: "4a46d417-45f9-421d-9e43-ab64ccca5c0d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:25:23.327204 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.327184    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk" (OuterVolumeSpecName: "kube-api-access-gd6lk") pod "4a46d417-45f9-421d-9e43-ab64ccca5c0d" (UID: "4a46d417-45f9-421d-9e43-ab64ccca5c0d"). InnerVolumeSpecName "kube-api-access-gd6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:25:23.425741 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.425710    2560 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4a46d417-45f9-421d-9e43-ab64ccca5c0d-config-file\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:25:23.425741 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:23.425741    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6lk\" (UniqueName: \"kubernetes.io/projected/4a46d417-45f9-421d-9e43-ab64ccca5c0d-kube-api-access-gd6lk\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:25:24.140710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:24.140688    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p"
Jul 23 09:25:24.140710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:24.140704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-tct4p" event={"ID":"4a46d417-45f9-421d-9e43-ab64ccca5c0d","Type":"ContainerDied","Data":"a9b0a07ea50c74e49644d6b2a4b9a3ad20c4a6462630a2c5d519a5c763f6be44"}
Jul 23 09:25:24.141180 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:24.140778    2560 scope.go:117] "RemoveContainer" containerID="7a333feaed39b4ed0cb3609fbb2ab0a54affcee4fd5a44b4c27bde15ae6e359b"
Jul 23 09:25:24.158968 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:24.158943    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:24.162961 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:24.162939    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-tct4p"]
Jul 23 09:25:25.912315 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:25.912286    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" path="/var/lib/kubelet/pods/4a46d417-45f9-421d-9e43-ab64ccca5c0d/volumes"
Jul 23 09:25:27.461803 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.461770    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"]
Jul 23 09:25:27.462341 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.462324    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" containerName="limitador"
Jul 23 09:25:27.462388 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.462344    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" containerName="limitador"
Jul 23 09:25:27.462434 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.462423    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a46d417-45f9-421d-9e43-ab64ccca5c0d" containerName="limitador"
Jul 23 09:25:27.467232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.467211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.469979 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.469952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 23 09:25:27.469979 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.469972    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 09:25:27.470194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.470103    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 09:25:27.471067 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.471047    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tbvsc\""
Jul 23 09:25:27.473439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.473416    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"]
Jul 23 09:25:27.555723 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.555700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2ab8-0198-446c-89b9-249020a9f62f-data\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.555830 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.555736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnz6j\" (UniqueName: \"kubernetes.io/projected/ee1e2ab8-0198-446c-89b9-249020a9f62f-kube-api-access-vnz6j\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.656830 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.656804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2ab8-0198-446c-89b9-249020a9f62f-data\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.656950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.656840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnz6j\" (UniqueName: \"kubernetes.io/projected/ee1e2ab8-0198-446c-89b9-249020a9f62f-kube-api-access-vnz6j\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.657181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.657165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee1e2ab8-0198-446c-89b9-249020a9f62f-data\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.671269 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.671243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnz6j\" (UniqueName: \"kubernetes.io/projected/ee1e2ab8-0198-446c-89b9-249020a9f62f-kube-api-access-vnz6j\") pod \"postgres-69fb4594cf-fk9b8\" (UID: \"ee1e2ab8-0198-446c-89b9-249020a9f62f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.777855 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.777791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:27.905585 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:27.905559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"]
Jul 23 09:25:27.907990 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:25:27.907960    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1e2ab8_0198_446c_89b9_249020a9f62f.slice/crio-6c06b18a9a86dda7a27cad09c2bf7541bed5f8a78b96e27291ace48ca5b8ca8d WatchSource:0}: Error finding container 6c06b18a9a86dda7a27cad09c2bf7541bed5f8a78b96e27291ace48ca5b8ca8d: Status 404 returned error can't find the container with id 6c06b18a9a86dda7a27cad09c2bf7541bed5f8a78b96e27291ace48ca5b8ca8d
Jul 23 09:25:28.158818 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:28.158788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8" event={"ID":"ee1e2ab8-0198-446c-89b9-249020a9f62f","Type":"ContainerStarted","Data":"6c06b18a9a86dda7a27cad09c2bf7541bed5f8a78b96e27291ace48ca5b8ca8d"}
Jul 23 09:25:33.186090 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:33.186028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8" event={"ID":"ee1e2ab8-0198-446c-89b9-249020a9f62f","Type":"ContainerStarted","Data":"f3215ebe787efef53f9b885fcc07856d6ba9b5240759a30c9043a336db916b64"}
Jul 23 09:25:33.186505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:33.186117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:33.203265 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:33.203218    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8" podStartSLOduration=1.56207838 podStartE2EDuration="6.203204324s" totalImagesPullingTime="4.641125944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:33.202290973 +0000 UTC m=+517.880508781" watchObservedRunningTime="2026-07-23 09:25:33.203204324 +0000 UTC m=+517.881422108"
Jul 23 09:25:39.220115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:39.220087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fk9b8"
Jul 23 09:25:46.581223 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.581192    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:25:46.607067 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.607040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:25:46.607219 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.607146    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:46.610972 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.610941    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 09:25:46.611193 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.611174    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nprx9\""
Jul 23 09:25:46.612020 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.612006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 09:25:46.708179 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.708155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:46.708276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.708219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkjc\" (UniqueName: \"kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:46.809062 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.809036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkjc\" (UniqueName: \"kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:46.809153 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.809089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:46.809208 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:25:46.809194    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 23 09:25:46.809274 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:25:46.809263    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert podName:a5d3f5e8-a746-4f8b-bc35-767da18d43c3 nodeName:}" failed. No retries permitted until 2026-07-23 09:25:47.309245009 +0000 UTC m=+531.987462789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert") pod "maas-controller-777776f6fc-5kdrs" (UID: "a5d3f5e8-a746-4f8b-bc35-767da18d43c3") : secret "maas-controller-webhook-cert" not found
Jul 23 09:25:46.820173 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:46.820148    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkjc\" (UniqueName: \"kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:47.312558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:47.312531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:47.312727 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:25:47.312652    2560 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 23 09:25:47.312727 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:25:47.312712    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert podName:a5d3f5e8-a746-4f8b-bc35-767da18d43c3 nodeName:}" failed. No retries permitted until 2026-07-23 09:25:48.312695424 +0000 UTC m=+532.990913192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert") pod "maas-controller-777776f6fc-5kdrs" (UID: "a5d3f5e8-a746-4f8b-bc35-767da18d43c3") : secret "maas-controller-webhook-cert" not found
Jul 23 09:25:48.321074 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:48.321043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:48.323232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:48.323212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"maas-controller-777776f6fc-5kdrs\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") " pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:48.419847 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:48.419821    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:48.546508 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:48.546484    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:25:48.548202 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:25:48.548170    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d3f5e8_a746_4f8b_bc35_767da18d43c3.slice/crio-18519283dea744894bc5432a3abba8723f98f613458e3acb617903d1a86c36ca WatchSource:0}: Error finding container 18519283dea744894bc5432a3abba8723f98f613458e3acb617903d1a86c36ca: Status 404 returned error can't find the container with id 18519283dea744894bc5432a3abba8723f98f613458e3acb617903d1a86c36ca
Jul 23 09:25:49.261282 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:49.261233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777776f6fc-5kdrs" event={"ID":"a5d3f5e8-a746-4f8b-bc35-767da18d43c3","Type":"ContainerStarted","Data":"18519283dea744894bc5432a3abba8723f98f613458e3acb617903d1a86c36ca"}
Jul 23 09:25:51.272211 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:51.272173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777776f6fc-5kdrs" event={"ID":"a5d3f5e8-a746-4f8b-bc35-767da18d43c3","Type":"ContainerStarted","Data":"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"}
Jul 23 09:25:51.272584 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:51.272234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:25:51.292191 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:25:51.292144    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777776f6fc-5kdrs" podStartSLOduration=3.154530289 podStartE2EDuration="5.292129341s" totalImagesPullingTime="2.137599052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:25:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:25:51.290544254 +0000 UTC m=+535.968762065" watchObservedRunningTime="2026-07-23 09:25:51.292129341 +0000 UTC m=+535.970347122"
Jul 23 09:26:01.579815 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.579778    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:26:01.580307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.580022    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-777776f6fc-5kdrs" podUID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" containerName="manager" containerID="cri-o://ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd" gracePeriod=30
Jul 23 09:26:01.581669 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.581646    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:26:01.818671 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.818638    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:26:01.921109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.921082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") pod \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") "
Jul 23 09:26:01.921294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.921205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpkjc\" (UniqueName: \"kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc\") pod \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\" (UID: \"a5d3f5e8-a746-4f8b-bc35-767da18d43c3\") "
Jul 23 09:26:01.923177 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.923150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a5d3f5e8-a746-4f8b-bc35-767da18d43c3" (UID: "a5d3f5e8-a746-4f8b-bc35-767da18d43c3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:26:01.923288 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:01.923212    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc" (OuterVolumeSpecName: "kube-api-access-tpkjc") pod "a5d3f5e8-a746-4f8b-bc35-767da18d43c3" (UID: "a5d3f5e8-a746-4f8b-bc35-767da18d43c3"). InnerVolumeSpecName "kube-api-access-tpkjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:26:02.023038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.023005    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-webhook-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:26:02.023171 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.023046    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkjc\" (UniqueName: \"kubernetes.io/projected/a5d3f5e8-a746-4f8b-bc35-767da18d43c3-kube-api-access-tpkjc\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:26:02.320208 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.320124    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" containerID="ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd" exitCode=0
Jul 23 09:26:02.320208 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.320174    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777776f6fc-5kdrs"
Jul 23 09:26:02.320451 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.320213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777776f6fc-5kdrs" event={"ID":"a5d3f5e8-a746-4f8b-bc35-767da18d43c3","Type":"ContainerDied","Data":"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"}
Jul 23 09:26:02.320451 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.320275    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777776f6fc-5kdrs" event={"ID":"a5d3f5e8-a746-4f8b-bc35-767da18d43c3","Type":"ContainerDied","Data":"18519283dea744894bc5432a3abba8723f98f613458e3acb617903d1a86c36ca"}
Jul 23 09:26:02.320451 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.320305    2560 scope.go:117] "RemoveContainer" containerID="ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"
Jul 23 09:26:02.329005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.328989    2560 scope.go:117] "RemoveContainer" containerID="ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"
Jul 23 09:26:02.329236 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:26:02.329215    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd\": container with ID starting with ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd not found: ID does not exist" containerID="ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"
Jul 23 09:26:02.329283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.329245    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd"} err="failed to get container status \"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd\": rpc error: code = NotFound desc = could not find container \"ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd\": container with ID starting with ad13197a4acd9951c1febc5f81570ba85d8f5779cb91978c90fde57c6a312edd not found: ID does not exist"
Jul 23 09:26:02.342046 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.342019    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:26:02.343729 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:02.343709    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-777776f6fc-5kdrs"]
Jul 23 09:26:03.913099 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:03.913065    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" path="/var/lib/kubelet/pods/a5d3f5e8-a746-4f8b-bc35-767da18d43c3/volumes"
Jul 23 09:26:23.221186 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.221151    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"]
Jul 23 09:26:23.221631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.221511    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" containerName="manager"
Jul 23 09:26:23.221631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.221521    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" containerName="manager"
Jul 23 09:26:23.221631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.221578    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d3f5e8-a746-4f8b-bc35-767da18d43c3" containerName="manager"
Jul 23 09:26:23.224423 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.224405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.227037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.227013    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c7gg4\""
Jul 23 09:26:23.227139 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.227084    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 09:26:23.227186 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.227156    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 09:26:23.227504 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.227488    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 09:26:23.236394 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.236371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"]
Jul 23 09:26:23.250896 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.250872    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-x2k4v"]
Jul 23 09:26:23.254120 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.254094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.268859 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.268834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-x2k4v"]
Jul 23 09:26:23.276027 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.276003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bef368c-87c7-495b-87d5-3a834ebb9e15-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.276113 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.276055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lr9b\" (UniqueName: \"kubernetes.io/projected/1bef368c-87c7-495b-87d5-3a834ebb9e15-kube-api-access-6lr9b\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.377205 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.377181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lr9b\" (UniqueName: \"kubernetes.io/projected/1bef368c-87c7-495b-87d5-3a834ebb9e15-kube-api-access-6lr9b\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.377398 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.377239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5584fdd3-e2a9-429e-a7b2-8367834a10fa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.377398 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.377264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsrl\" (UniqueName: \"kubernetes.io/projected/5584fdd3-e2a9-429e-a7b2-8367834a10fa-kube-api-access-9nsrl\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.377398 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.377281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bef368c-87c7-495b-87d5-3a834ebb9e15-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.377939 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.377903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bef368c-87c7-495b-87d5-3a834ebb9e15-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.385750 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.385733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lr9b\" (UniqueName: \"kubernetes.io/projected/1bef368c-87c7-495b-87d5-3a834ebb9e15-kube-api-access-6lr9b\") pod \"payload-pre-processing-7677b774bb-wrrg6\" (UID: \"1bef368c-87c7-495b-87d5-3a834ebb9e15\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.477842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.477780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5584fdd3-e2a9-429e-a7b2-8367834a10fa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.477842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.477824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsrl\" (UniqueName: \"kubernetes.io/projected/5584fdd3-e2a9-429e-a7b2-8367834a10fa-kube-api-access-9nsrl\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.478488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.478468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5584fdd3-e2a9-429e-a7b2-8367834a10fa-plugins-config-volume\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.486322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.486302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsrl\" (UniqueName: \"kubernetes.io/projected/5584fdd3-e2a9-429e-a7b2-8367834a10fa-kube-api-access-9nsrl\") pod \"payload-processing-5bf47cd988-x2k4v\" (UID: \"5584fdd3-e2a9-429e-a7b2-8367834a10fa\") " pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.536198 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.536176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:23.563663 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.563640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:23.678720 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.678674    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"]
Jul 23 09:26:23.681012 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:26:23.680971    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef368c_87c7_495b_87d5_3a834ebb9e15.slice/crio-c513f94ac44fc17153dc74811fd1d87f5d50be437256797946e2ea947eedd4c7 WatchSource:0}: Error finding container c513f94ac44fc17153dc74811fd1d87f5d50be437256797946e2ea947eedd4c7: Status 404 returned error can't find the container with id c513f94ac44fc17153dc74811fd1d87f5d50be437256797946e2ea947eedd4c7
Jul 23 09:26:23.701875 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:23.701853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-x2k4v"]
Jul 23 09:26:23.703652 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:26:23.703629    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5584fdd3_e2a9_429e_a7b2_8367834a10fa.slice/crio-44b3e997f7c8ab140e9da53b4ebb29c7e8218f6058d039b2e0af62345237824d WatchSource:0}: Error finding container 44b3e997f7c8ab140e9da53b4ebb29c7e8218f6058d039b2e0af62345237824d: Status 404 returned error can't find the container with id 44b3e997f7c8ab140e9da53b4ebb29c7e8218f6058d039b2e0af62345237824d
Jul 23 09:26:24.429510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:24.429434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v" event={"ID":"5584fdd3-e2a9-429e-a7b2-8367834a10fa","Type":"ContainerStarted","Data":"44b3e997f7c8ab140e9da53b4ebb29c7e8218f6058d039b2e0af62345237824d"}
Jul 23 09:26:24.432765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:24.432731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6" event={"ID":"1bef368c-87c7-495b-87d5-3a834ebb9e15","Type":"ContainerStarted","Data":"c513f94ac44fc17153dc74811fd1d87f5d50be437256797946e2ea947eedd4c7"}
Jul 23 09:26:26.444797 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.444768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v" event={"ID":"5584fdd3-e2a9-429e-a7b2-8367834a10fa","Type":"ContainerStarted","Data":"3a5cf7d089ae610bb202b10606e7fda5ebeebd813bdf357a4c4eb774e2b2e5ad"}
Jul 23 09:26:26.445264 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.444890    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:26.446455 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.446434    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6" event={"ID":"1bef368c-87c7-495b-87d5-3a834ebb9e15","Type":"ContainerStarted","Data":"625ed7ee329c83208f63ea1a814a5cd53d102aed33deb4a5b64ed4d0e332fa72"}
Jul 23 09:26:26.446563 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.446479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:26.466550 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.466509    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v" podStartSLOduration=1.077142789 podStartE2EDuration="3.466496169s" totalImagesPullingTime="2.38935338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:26:26.465267779 +0000 UTC m=+571.143485573" watchObservedRunningTime="2026-07-23 09:26:26.466496169 +0000 UTC m=+571.144713983"
Jul 23 09:26:26.483312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:26.483264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6" podStartSLOduration=1.077870302 podStartE2EDuration="3.483252494s" totalImagesPullingTime="2.405382192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:26:26.482433484 +0000 UTC m=+571.160651267" watchObservedRunningTime="2026-07-23 09:26:26.483252494 +0000 UTC m=+571.161470277"
Jul 23 09:26:37.453691 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:37.453662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-x2k4v"
Jul 23 09:26:37.454223 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:37.453735    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-wrrg6"
Jul 23 09:26:59.168011 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.167969    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"]
Jul 23 09:26:59.170633 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.170609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.173419 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.173398    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 09:26:59.174535 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.174514    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 09:26:59.174641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.174534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hw9h8\""
Jul 23 09:26:59.174641 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.174555    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 09:26:59.185403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.185371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"]
Jul 23 09:26:59.260346 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.260346 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.260528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.260528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2t6q\" (UniqueName: \"kubernetes.io/projected/eb18cf27-205e-4c4f-aad1-b5459a521ee8-kube-api-access-m2t6q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.260638 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.260638 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.260596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361370 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361470 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361470 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361470 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361470 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361675 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2t6q\" (UniqueName: \"kubernetes.io/projected/eb18cf27-205e-4c4f-aad1-b5459a521ee8-kube-api-access-m2t6q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361860 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.361934 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.361875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.363517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.363499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb18cf27-205e-4c4f-aad1-b5459a521ee8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.363770 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.363753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb18cf27-205e-4c4f-aad1-b5459a521ee8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.370054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.370033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2t6q\" (UniqueName: \"kubernetes.io/projected/eb18cf27-205e-4c4f-aad1-b5459a521ee8-kube-api-access-m2t6q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wtcvb\" (UID: \"eb18cf27-205e-4c4f-aad1-b5459a521ee8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.487242 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.487184    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:26:59.618533 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.618507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"]
Jul 23 09:26:59.621115 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:26:59.621088    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb18cf27_205e_4c4f_aad1_b5459a521ee8.slice/crio-2cd59653abcc2bf5efcafc1cc86bfd84ac844091cbf2ed42bddb346f081f137a WatchSource:0}: Error finding container 2cd59653abcc2bf5efcafc1cc86bfd84ac844091cbf2ed42bddb346f081f137a: Status 404 returned error can't find the container with id 2cd59653abcc2bf5efcafc1cc86bfd84ac844091cbf2ed42bddb346f081f137a
Jul 23 09:26:59.622871 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:26:59.622851    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:27:00.605449 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:00.605404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb" event={"ID":"eb18cf27-205e-4c4f-aad1-b5459a521ee8","Type":"ContainerStarted","Data":"2cd59653abcc2bf5efcafc1cc86bfd84ac844091cbf2ed42bddb346f081f137a"}
Jul 23 09:27:01.611623 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:01.611586    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb" event={"ID":"eb18cf27-205e-4c4f-aad1-b5459a521ee8","Type":"ContainerStarted","Data":"11c1ce3f4e15593aecd2d756aae0225790e8ae75ec495695e68b4c357d7b5a16"}
Jul 23 09:27:01.628896 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:01.628852    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb" podStartSLOduration=1.527588524 podStartE2EDuration="2.628837095s" totalImagesPullingTime="1.101248571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:27:01.628160499 +0000 UTC m=+606.306378296" watchObservedRunningTime="2026-07-23 09:27:01.628837095 +0000 UTC m=+606.307054879"
Jul 23 09:27:04.571168 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.571133    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"]
Jul 23 09:27:04.573498 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.573481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.576711 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.576687    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 09:27:04.587941 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.587903    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"]
Jul 23 09:27:04.608832 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.608812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.608969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.608839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4898197e-f975-4dbd-9525-e557d8081c83-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.608969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.608873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.608969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.608904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.608969 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.608947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.609133 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.609012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbf72\" (UniqueName: \"kubernetes.io/projected/4898197e-f975-4dbd-9525-e557d8081c83-kube-api-access-lbf72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710220 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710220 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbf72\" (UniqueName: \"kubernetes.io/projected/4898197e-f975-4dbd-9525-e557d8081c83-kube-api-access-lbf72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710220 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710421 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4898197e-f975-4dbd-9525-e557d8081c83-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710421 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710691 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710691 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.710887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.710685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.712661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.712631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4898197e-f975-4dbd-9525-e557d8081c83-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.712661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.712650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4898197e-f975-4dbd-9525-e557d8081c83-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.721799 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.721778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbf72\" (UniqueName: \"kubernetes.io/projected/4898197e-f975-4dbd-9525-e557d8081c83-kube-api-access-lbf72\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mf67f\" (UID: \"4898197e-f975-4dbd-9525-e557d8081c83\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:04.884062 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:04.884007    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:05.011070 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:05.011008    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"]
Jul 23 09:27:05.012037 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:27:05.012014    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4898197e_f975_4dbd_9525_e557d8081c83.slice/crio-5c76e9f2ea718e0f398b9c7542faa21c0c75004b154bd88e3bcb087fe589d192 WatchSource:0}: Error finding container 5c76e9f2ea718e0f398b9c7542faa21c0c75004b154bd88e3bcb087fe589d192: Status 404 returned error can't find the container with id 5c76e9f2ea718e0f398b9c7542faa21c0c75004b154bd88e3bcb087fe589d192
Jul 23 09:27:05.633209 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:05.633111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f" event={"ID":"4898197e-f975-4dbd-9525-e557d8081c83","Type":"ContainerStarted","Data":"dd8d69a99968ab6e732a3732324ebd7447474f9ed579e71f480ead7d89bf7b20"}
Jul 23 09:27:05.633209 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:05.633178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f" event={"ID":"4898197e-f975-4dbd-9525-e557d8081c83","Type":"ContainerStarted","Data":"5c76e9f2ea718e0f398b9c7542faa21c0c75004b154bd88e3bcb087fe589d192"}
Jul 23 09:27:05.651187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:05.651141    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f" podStartSLOduration=1.44714944 podStartE2EDuration="1.651127927s" totalImagesPullingTime="203.978487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:27:05.649433264 +0000 UTC m=+610.327651063" watchObservedRunningTime="2026-07-23 09:27:05.651127927 +0000 UTC m=+610.329345710"
Jul 23 09:27:09.487303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:09.487269    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:27:09.487679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:09.487322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:27:09.499745 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:09.499720    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:27:09.663558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:09.663529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wtcvb"
Jul 23 09:27:14.884652 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:14.884618    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:14.885046 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:14.884674    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:14.896851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:14.896829    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:27:15.692960 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:27:15.692936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mf67f"
Jul 23 09:30:00.146984 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.146943    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"]
Jul 23 09:30:00.150454 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.150437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"
Jul 23 09:30:00.153409 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.153389    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\""
Jul 23 09:30:00.158401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.158336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"]
Jul 23 09:30:00.162579 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.162551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"
Jul 23 09:30:00.516789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:00.516765    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"]
Jul 23 09:30:00.518670 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:30:00.518642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75fa870_96d6_4035_b54e_91cdeead9da0.slice/crio-a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464 WatchSource:0}: Error finding container a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464: Status 404 returned error can't find the container with id a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464
Jul 23 09:30:01.449443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:01.449406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh" event={"ID":"b75fa870-96d6-4035-b54e-91cdeead9da0","Type":"ContainerStarted","Data":"a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464"}
Jul 23 09:30:04.465512 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:04.465480    2560 generic.go:356] "Generic (PLEG): container finished" podID="b75fa870-96d6-4035-b54e-91cdeead9da0" containerID="3467ec40f1078b7ca89aa46238bb050368ca4bdd20aa5f5a04c7aa031c843682" exitCode=0
Jul 23 09:30:04.465887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:04.465518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh" event={"ID":"b75fa870-96d6-4035-b54e-91cdeead9da0","Type":"ContainerDied","Data":"3467ec40f1078b7ca89aa46238bb050368ca4bdd20aa5f5a04c7aa031c843682"}
Jul 23 09:30:05.609350 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:05.609325    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"
Jul 23 09:30:06.475976 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:06.475946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh"
Jul 23 09:30:06.476191 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:06.475971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746650-5plfh" event={"ID":"b75fa870-96d6-4035-b54e-91cdeead9da0","Type":"ContainerDied","Data":"a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464"}
Jul 23 09:30:06.476191 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:06.476020    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10e538d11c3e05c1b3c5365efb8dc83a80208396813247be9b4ba767d632464"
Jul 23 09:30:25.325826 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.325745    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:25.326221 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.326141    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75fa870-96d6-4035-b54e-91cdeead9da0" containerName="cleanup"
Jul 23 09:30:25.326221 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.326152    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75fa870-96d6-4035-b54e-91cdeead9da0" containerName="cleanup"
Jul 23 09:30:25.326294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.326225    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75fa870-96d6-4035-b54e-91cdeead9da0" containerName="cleanup"
Jul 23 09:30:25.329124 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.329108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.334019 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.333993    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nprx9\""
Jul 23 09:30:25.334019 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.334015    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 09:30:25.334217 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.333993    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 09:30:25.344967 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.344943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:25.495119 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.495082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.495289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.495144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp5d\" (UniqueName: \"kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.596369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.596290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.596369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.596338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp5d\" (UniqueName: \"kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.598791 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.598766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.608698 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.608675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp5d\" (UniqueName: \"kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d\") pod \"maas-controller-68c9f8fb88-cswbd\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") " pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.640729 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.640701    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:25.795253 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:25.795228    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:25.797635 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:30:25.797607    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74189561_0cbe_414e_873e_b817951854f6.slice/crio-403f3bdc11d641a0236d4ec339218d92c3c0ca667d80e412a5191ca48cf5a796 WatchSource:0}: Error finding container 403f3bdc11d641a0236d4ec339218d92c3c0ca667d80e412a5191ca48cf5a796: Status 404 returned error can't find the container with id 403f3bdc11d641a0236d4ec339218d92c3c0ca667d80e412a5191ca48cf5a796
Jul 23 09:30:26.575752 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:26.575670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" event={"ID":"74189561-0cbe-414e-873e-b817951854f6","Type":"ContainerStarted","Data":"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"}
Jul 23 09:30:26.575752 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:26.575729    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" event={"ID":"74189561-0cbe-414e-873e-b817951854f6","Type":"ContainerStarted","Data":"403f3bdc11d641a0236d4ec339218d92c3c0ca667d80e412a5191ca48cf5a796"}
Jul 23 09:30:26.576159 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:26.575757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:26.616536 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:26.616470    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" podStartSLOduration=1.113614739 podStartE2EDuration="1.616449807s" totalImagesPullingTime="502.835068ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:30:26.615241177 +0000 UTC m=+811.293458962" watchObservedRunningTime="2026-07-23 09:30:26.616449807 +0000 UTC m=+811.294667593"
Jul 23 09:30:37.582987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:37.582956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:42.285044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.285006    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:42.285553 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.285243    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" podUID="74189561-0cbe-414e-873e-b817951854f6" containerName="manager" containerID="cri-o://0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a" gracePeriod=30
Jul 23 09:30:42.542610 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.542556    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:42.648177 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.648141    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gp5d\" (UniqueName: \"kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d\") pod \"74189561-0cbe-414e-873e-b817951854f6\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") "
Jul 23 09:30:42.648359 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.648342    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert\") pod \"74189561-0cbe-414e-873e-b817951854f6\" (UID: \"74189561-0cbe-414e-873e-b817951854f6\") "
Jul 23 09:30:42.650384 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.650360    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d" (OuterVolumeSpecName: "kube-api-access-5gp5d") pod "74189561-0cbe-414e-873e-b817951854f6" (UID: "74189561-0cbe-414e-873e-b817951854f6"). InnerVolumeSpecName "kube-api-access-5gp5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:30:42.650504 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.650414    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "74189561-0cbe-414e-873e-b817951854f6" (UID: "74189561-0cbe-414e-873e-b817951854f6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:30:42.653729 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.653699    2560 generic.go:356] "Generic (PLEG): container finished" podID="74189561-0cbe-414e-873e-b817951854f6" containerID="0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a" exitCode=0
Jul 23 09:30:42.653816 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.653744    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-cswbd"
Jul 23 09:30:42.653816 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.653773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" event={"ID":"74189561-0cbe-414e-873e-b817951854f6","Type":"ContainerDied","Data":"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"}
Jul 23 09:30:42.653889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.653831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-cswbd" event={"ID":"74189561-0cbe-414e-873e-b817951854f6","Type":"ContainerDied","Data":"403f3bdc11d641a0236d4ec339218d92c3c0ca667d80e412a5191ca48cf5a796"}
Jul 23 09:30:42.653889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.653853    2560 scope.go:117] "RemoveContainer" containerID="0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"
Jul 23 09:30:42.668434 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.668417    2560 scope.go:117] "RemoveContainer" containerID="0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"
Jul 23 09:30:42.668662 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:30:42.668647    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a\": container with ID starting with 0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a not found: ID does not exist" containerID="0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"
Jul 23 09:30:42.668712 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.668668    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a"} err="failed to get container status \"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a\": rpc error: code = NotFound desc = could not find container \"0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a\": container with ID starting with 0d7febcea76e429b0451b7a0c2d447f9af375e60d203976bed377434e8a7b71a not found: ID does not exist"
Jul 23 09:30:42.697847 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.697817    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:42.699538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.699517    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-cswbd"]
Jul 23 09:30:42.749539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.749510    2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74189561-0cbe-414e-873e-b817951854f6-webhook-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:30:42.749539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:42.749536    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp5d\" (UniqueName: \"kubernetes.io/projected/74189561-0cbe-414e-873e-b817951854f6-kube-api-access-5gp5d\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:30:43.559128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.559094    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-8m69g"]
Jul 23 09:30:43.559499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.559484    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74189561-0cbe-414e-873e-b817951854f6" containerName="manager"
Jul 23 09:30:43.559499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.559495    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="74189561-0cbe-414e-873e-b817951854f6" containerName="manager"
Jul 23 09:30:43.559573 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.559556    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="74189561-0cbe-414e-873e-b817951854f6" containerName="manager"
Jul 23 09:30:43.564133 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.564112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.570866 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.570848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 09:30:43.570980 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.570958    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nprx9\""
Jul 23 09:30:43.571710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.571693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 09:30:43.612121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.612093    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-8m69g"]
Jul 23 09:30:43.657401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.657368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27950d6c-29fa-47e0-b465-8507f8a016c6-webhook-cert\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.657549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.657415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rtl\" (UniqueName: \"kubernetes.io/projected/27950d6c-29fa-47e0-b465-8507f8a016c6-kube-api-access-28rtl\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.758889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.758852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27950d6c-29fa-47e0-b465-8507f8a016c6-webhook-cert\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.759065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.758896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rtl\" (UniqueName: \"kubernetes.io/projected/27950d6c-29fa-47e0-b465-8507f8a016c6-kube-api-access-28rtl\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.761270 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.761249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27950d6c-29fa-47e0-b465-8507f8a016c6-webhook-cert\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.770572 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.770551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rtl\" (UniqueName: \"kubernetes.io/projected/27950d6c-29fa-47e0-b465-8507f8a016c6-kube-api-access-28rtl\") pod \"maas-controller-68c9f8fb88-8m69g\" (UID: \"27950d6c-29fa-47e0-b465-8507f8a016c6\") " pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.874191 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.874093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:43.913750 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:43.913710    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74189561-0cbe-414e-873e-b817951854f6" path="/var/lib/kubelet/pods/74189561-0cbe-414e-873e-b817951854f6/volumes"
Jul 23 09:30:44.030712 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:44.030661    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c9f8fb88-8m69g"]
Jul 23 09:30:44.034598 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:30:44.034569    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27950d6c_29fa_47e0_b465_8507f8a016c6.slice/crio-e6eab5bc701770ac95730f0cade2df6e10ee91ed8d6e6b72d90b1a35dfa04505 WatchSource:0}: Error finding container e6eab5bc701770ac95730f0cade2df6e10ee91ed8d6e6b72d90b1a35dfa04505: Status 404 returned error can't find the container with id e6eab5bc701770ac95730f0cade2df6e10ee91ed8d6e6b72d90b1a35dfa04505
Jul 23 09:30:44.666350 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:44.666312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-8m69g" event={"ID":"27950d6c-29fa-47e0-b465-8507f8a016c6","Type":"ContainerStarted","Data":"f18f65d07d087fbb32f1e355b0219b9cab9d67a92044033cd5c68ea303f8d01b"}
Jul 23 09:30:44.666820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:44.666391    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c9f8fb88-8m69g" event={"ID":"27950d6c-29fa-47e0-b465-8507f8a016c6","Type":"ContainerStarted","Data":"e6eab5bc701770ac95730f0cade2df6e10ee91ed8d6e6b72d90b1a35dfa04505"}
Jul 23 09:30:44.666820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:44.666439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:30:44.693538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:44.693480    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68c9f8fb88-8m69g" podStartSLOduration=1.3510602409999999 podStartE2EDuration="1.693464422s" totalImagesPullingTime="342.404181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:30:44.691116348 +0000 UTC m=+829.369334131" watchObservedRunningTime="2026-07-23 09:30:44.693464422 +0000 UTC m=+829.371682204"
Jul 23 09:30:55.674179 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:30:55.674144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c9f8fb88-8m69g"
Jul 23 09:35:32.884772 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:32.884739    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:35:32.885417 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:32.884946    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" podUID="734baa38-779b-4815-9050-bf348901823e" containerName="manager" containerID="cri-o://611867224baebb251f9b1bce43120dde498fab67bfcb845ba2ca7eed63365dce" gracePeriod=10
Jul 23 09:35:37.087037 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.087000    2560 generic.go:356] "Generic (PLEG): container finished" podID="734baa38-779b-4815-9050-bf348901823e" containerID="611867224baebb251f9b1bce43120dde498fab67bfcb845ba2ca7eed63365dce" exitCode=0
Jul 23 09:35:37.087426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.087076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" event={"ID":"734baa38-779b-4815-9050-bf348901823e","Type":"ContainerDied","Data":"611867224baebb251f9b1bce43120dde498fab67bfcb845ba2ca7eed63365dce"}
Jul 23 09:35:37.140069 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.140040    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:35:37.302169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.302087    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume\") pod \"734baa38-779b-4815-9050-bf348901823e\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") "
Jul 23 09:35:37.302169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.302142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d85n\" (UniqueName: \"kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n\") pod \"734baa38-779b-4815-9050-bf348901823e\" (UID: \"734baa38-779b-4815-9050-bf348901823e\") "
Jul 23 09:35:37.302548 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.302510    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "734baa38-779b-4815-9050-bf348901823e" (UID: "734baa38-779b-4815-9050-bf348901823e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:35:37.304431 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.304408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n" (OuterVolumeSpecName: "kube-api-access-9d85n") pod "734baa38-779b-4815-9050-bf348901823e" (UID: "734baa38-779b-4815-9050-bf348901823e"). InnerVolumeSpecName "kube-api-access-9d85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:35:37.403061 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.403028    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/734baa38-779b-4815-9050-bf348901823e-extensions-socket-volume\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:35:37.403061 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:37.403062    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d85n\" (UniqueName: \"kubernetes.io/projected/734baa38-779b-4815-9050-bf348901823e-kube-api-access-9d85n\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:35:38.093675 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:38.093642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"
Jul 23 09:35:38.094125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:38.093642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg" event={"ID":"734baa38-779b-4815-9050-bf348901823e","Type":"ContainerDied","Data":"798ce4c92dc08713787db91754c51ee71bd66c329bd585cdc407532e7d304636"}
Jul 23 09:35:38.094125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:38.093773    2560 scope.go:117] "RemoveContainer" containerID="611867224baebb251f9b1bce43120dde498fab67bfcb845ba2ca7eed63365dce"
Jul 23 09:35:38.116357 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:38.116334    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:35:38.122376 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:38.122348    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-smxbg"]
Jul 23 09:35:39.912891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:35:39.912857    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734baa38-779b-4815-9050-bf348901823e" path="/var/lib/kubelet/pods/734baa38-779b-4815-9050-bf348901823e/volumes"
Jul 23 09:36:34.072134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.072052    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"]
Jul 23 09:36:34.072727 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.072621    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734baa38-779b-4815-9050-bf348901823e" containerName="manager"
Jul 23 09:36:34.072727 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.072639    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="734baa38-779b-4815-9050-bf348901823e" containerName="manager"
Jul 23 09:36:34.072825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.072756    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="734baa38-779b-4815-9050-bf348901823e" containerName="manager"
Jul 23 09:36:34.075885 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.075868    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.080513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.080491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8kkf8\""
Jul 23 09:36:34.105838 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.105817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"]
Jul 23 09:36:34.181156 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.181128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.181332 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.181167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv5b\" (UniqueName: \"kubernetes.io/projected/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-kube-api-access-bvv5b\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.282519 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.282489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.282653 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.282530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv5b\" (UniqueName: \"kubernetes.io/projected/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-kube-api-access-bvv5b\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.282962 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.282940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.292333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.292311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv5b\" (UniqueName: \"kubernetes.io/projected/d83f55d6-df56-4db5-9a46-df7abbbe1c0a-kube-api-access-bvv5b\") pod \"kuadrant-operator-controller-manager-86878c769f-7mzw2\" (UID: \"d83f55d6-df56-4db5-9a46-df7abbbe1c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.387471 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.387409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:34.533568 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:36:34.533527    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83f55d6_df56_4db5_9a46_df7abbbe1c0a.slice/crio-5cf40242942ffb8175b3d3c42353001c787c067ff3611f7325c6a8d2b7f755dc WatchSource:0}: Error finding container 5cf40242942ffb8175b3d3c42353001c787c067ff3611f7325c6a8d2b7f755dc: Status 404 returned error can't find the container with id 5cf40242942ffb8175b3d3c42353001c787c067ff3611f7325c6a8d2b7f755dc
Jul 23 09:36:34.534042 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:34.534020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"]
Jul 23 09:36:35.363124 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:35.363092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2" event={"ID":"d83f55d6-df56-4db5-9a46-df7abbbe1c0a","Type":"ContainerStarted","Data":"4a3e542dcd6d8f0bb127530c1a23fd77b571a326677964d56c941e2d1917f7d3"}
Jul 23 09:36:35.363527 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:35.363143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2" event={"ID":"d83f55d6-df56-4db5-9a46-df7abbbe1c0a","Type":"ContainerStarted","Data":"5cf40242942ffb8175b3d3c42353001c787c067ff3611f7325c6a8d2b7f755dc"}
Jul 23 09:36:35.363527 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:35.363192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:36:35.393558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:35.390352    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2" podStartSLOduration=1.390335242 podStartE2EDuration="1.390335242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:36:35.385896805 +0000 UTC m=+1180.064114590" watchObservedRunningTime="2026-07-23 09:36:35.390335242 +0000 UTC m=+1180.068553026"
Jul 23 09:36:46.369894 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:36:46.369863    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7mzw2"
Jul 23 09:39:28.996593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:28.996521    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:39:28.999901 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:28.999884    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.002971 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.002951    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4f4177bf-kserve-self-signed-certs\""
Jul 23 09:39:29.020388 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.020368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:39:29.175432 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.175597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.175597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.175597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.175597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwl6\" (UniqueName: \"kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.175753 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.175635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.276970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277043 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277268 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwl6\" (UniqueName: \"kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277268 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277268 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277391 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277436 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.277472 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.277431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.279154 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.279136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.279445 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.279427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.285193 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.285170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwl6\" (UniqueName: \"kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6\") pod \"e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") " pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.308680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.308661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:29.444397 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:29.444370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:39:29.446881 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:39:29.446853    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d964b3_827e_4a96_b0d6_507021f6ef7b.slice/crio-c5fdf427efcc3725da010121432941797f3cb1d2e14d2bdebbf12f7f7dd0c0ba WatchSource:0}: Error finding container c5fdf427efcc3725da010121432941797f3cb1d2e14d2bdebbf12f7f7dd0c0ba: Status 404 returned error can't find the container with id c5fdf427efcc3725da010121432941797f3cb1d2e14d2bdebbf12f7f7dd0c0ba
Jul 23 09:39:30.223173 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:30.223132    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" event={"ID":"f9d964b3-827e-4a96-b0d6-507021f6ef7b","Type":"ContainerStarted","Data":"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"}
Jul 23 09:39:30.223618 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:30.223180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" event={"ID":"f9d964b3-827e-4a96-b0d6-507021f6ef7b","Type":"ContainerStarted","Data":"c5fdf427efcc3725da010121432941797f3cb1d2e14d2bdebbf12f7f7dd0c0ba"}
Jul 23 09:39:30.240699 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:30.240648    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" podStartSLOduration=2.240631116 podStartE2EDuration="2.240631116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:39:30.240425703 +0000 UTC m=+1354.918643487" watchObservedRunningTime="2026-07-23 09:39:30.240631116 +0000 UTC m=+1354.918848901"
Jul 23 09:39:39.309044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:39.308963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:39.309044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:39.309010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:39.321319 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:39.321296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:40.281416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:40.281387    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:39:45.379136 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.379099    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:39:45.383050 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.383027    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.385659 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.385641    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4f4177bf-kserve-self-signed-certs\""
Jul 23 09:39:45.393697 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.393674    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:39:45.411665 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.411751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411675    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzzhz\" (UniqueName: \"kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.411801 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.411842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.411895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.411956 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.411932    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512615 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.512761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.512720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzzhz\" (UniqueName: \"kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.513060 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.513028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.513125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.513072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.513125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.513102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.515036 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.515003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.515338 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.515319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.525251 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.525226    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzzhz\" (UniqueName: \"kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz\") pod \"e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") " pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.693638 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.693607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:45.831387 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:45.831354    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:39:45.834238 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:39:45.834211    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04478882_e34f_4b60_9a94_2a60fbfa7979.slice/crio-e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9 WatchSource:0}: Error finding container e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9: Status 404 returned error can't find the container with id e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9
Jul 23 09:39:46.300426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:46.300384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" event={"ID":"04478882-e34f-4b60-9a94-2a60fbfa7979","Type":"ContainerStarted","Data":"0eeab33630f479926a29297c3f5a61220e0992141d1d4f0263fb2020e70829b5"}
Jul 23 09:39:46.300624 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:46.300436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" event={"ID":"04478882-e34f-4b60-9a94-2a60fbfa7979","Type":"ContainerStarted","Data":"e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9"}
Jul 23 09:39:46.319118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:46.319069    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" podStartSLOduration=1.31905106 podStartE2EDuration="1.31905106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:39:46.317362515 +0000 UTC m=+1370.995580300" watchObservedRunningTime="2026-07-23 09:39:46.31905106 +0000 UTC m=+1370.997268843"
Jul 23 09:39:55.694165 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:55.694130    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:55.694165 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:55.694174    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:55.706496 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:55.706473    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:39:56.360564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:39:56.360537    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:40:36.295133 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:36.295098    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:40:37.365616 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:37.365586    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:40:51.316182 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.316062    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" podUID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" containerName="main" containerID="cri-o://4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2" gracePeriod=45
Jul 23 09:40:51.570203 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.570145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:40:51.619307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.619266    2560 generic.go:356] "Generic (PLEG): container finished" podID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" containerID="4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2" exitCode=0
Jul 23 09:40:51.619307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.619301    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"
Jul 23 09:40:51.619551 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.619326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" event={"ID":"f9d964b3-827e-4a96-b0d6-507021f6ef7b","Type":"ContainerDied","Data":"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"}
Jul 23 09:40:51.619551 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.619375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b" event={"ID":"f9d964b3-827e-4a96-b0d6-507021f6ef7b","Type":"ContainerDied","Data":"c5fdf427efcc3725da010121432941797f3cb1d2e14d2bdebbf12f7f7dd0c0ba"}
Jul 23 09:40:51.619551 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.619397    2560 scope.go:117] "RemoveContainer" containerID="4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"
Jul 23 09:40:51.627861 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.627841    2560 scope.go:117] "RemoveContainer" containerID="4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"
Jul 23 09:40:51.628133 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:40:51.628114    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2\": container with ID starting with 4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2 not found: ID does not exist" containerID="4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"
Jul 23 09:40:51.628194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.628146    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2"} err="failed to get container status \"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2\": rpc error: code = NotFound desc = could not find container \"4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2\": container with ID starting with 4417a04155e186786350bee548bc4649b7b9651b501d9df64a83358e6182bac2 not found: ID does not exist"
Jul 23 09:40:51.673389 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673523 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673523 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673449    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673523 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673471    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673523 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfwl6\" (UniqueName: \"kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir\") pod \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\" (UID: \"f9d964b3-827e-4a96-b0d6-507021f6ef7b\") "
Jul 23 09:40:51.673734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:51.673845 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673738    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home" (OuterVolumeSpecName: "home") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:51.673897 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673849    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:51.673897 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673880    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.674007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.673901    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.675534 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.675509    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:40:51.675609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.675593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm" (OuterVolumeSpecName: "dshm") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:51.675673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.675646    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6" (OuterVolumeSpecName: "kube-api-access-bfwl6") pod "f9d964b3-827e-4a96-b0d6-507021f6ef7b" (UID: "f9d964b3-827e-4a96-b0d6-507021f6ef7b"). InnerVolumeSpecName "kube-api-access-bfwl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:40:51.775258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.775223    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.775258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.775254    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwl6\" (UniqueName: \"kubernetes.io/projected/f9d964b3-827e-4a96-b0d6-507021f6ef7b-kube-api-access-bfwl6\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.775258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.775264    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.775473 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.775272    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d964b3-827e-4a96-b0d6-507021f6ef7b-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:51.935074 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.935044    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:40:51.939231 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:51.939209    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4f4177bf-kserve-5cb86cbc47-wgd5b"]
Jul 23 09:40:52.386006 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.385890    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" podUID="04478882-e34f-4b60-9a94-2a60fbfa7979" containerName="main" containerID="cri-o://0eeab33630f479926a29297c3f5a61220e0992141d1d4f0263fb2020e70829b5" gracePeriod=45
Jul 23 09:40:52.626044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.626017    2560 generic.go:356] "Generic (PLEG): container finished" podID="04478882-e34f-4b60-9a94-2a60fbfa7979" containerID="0eeab33630f479926a29297c3f5a61220e0992141d1d4f0263fb2020e70829b5" exitCode=0
Jul 23 09:40:52.626157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.626101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" event={"ID":"04478882-e34f-4b60-9a94-2a60fbfa7979","Type":"ContainerDied","Data":"0eeab33630f479926a29297c3f5a61220e0992141d1d4f0263fb2020e70829b5"}
Jul 23 09:40:52.626157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.626149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l" event={"ID":"04478882-e34f-4b60-9a94-2a60fbfa7979","Type":"ContainerDied","Data":"e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9"}
Jul 23 09:40:52.626270 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.626168    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6b20267f542b60cce0b4460f12878938615c529974cd3e950ca30db25a90dc9"
Jul 23 09:40:52.638235 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.638183    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:40:52.683194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683242    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzzhz\" (UniqueName: \"kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683345 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm\") pod \"04478882-e34f-4b60-9a94-2a60fbfa7979\" (UID: \"04478882-e34f-4b60-9a94-2a60fbfa7979\") "
Jul 23 09:40:52.683613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683500    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache" (OuterVolumeSpecName: "model-cache") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:52.683613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:52.683613 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.683567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home" (OuterVolumeSpecName: "home") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:52.685460 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.685437    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:40:52.685565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.685477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm" (OuterVolumeSpecName: "dshm") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:40:52.685565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.685502    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz" (OuterVolumeSpecName: "kube-api-access-wzzhz") pod "04478882-e34f-4b60-9a94-2a60fbfa7979" (UID: "04478882-e34f-4b60-9a94-2a60fbfa7979"). InnerVolumeSpecName "kube-api-access-wzzhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:40:52.784479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784452    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04478882-e34f-4b60-9a94-2a60fbfa7979-tls-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:52.784479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784478    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-model-cache\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:52.784635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784487    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-tmp-dir\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:52.784635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784495    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzzhz\" (UniqueName: \"kubernetes.io/projected/04478882-e34f-4b60-9a94-2a60fbfa7979-kube-api-access-wzzhz\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:52.784635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784503    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-home\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:52.784635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:52.784511    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04478882-e34f-4b60-9a94-2a60fbfa7979-dshm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:40:53.633717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:53.633681    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"
Jul 23 09:40:53.663625 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:53.663591    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:40:53.668178 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:53.668152    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4f4177bf-kserve-6645b6649f-5xp8l"]
Jul 23 09:40:53.912365 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:53.912334    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04478882-e34f-4b60-9a94-2a60fbfa7979" path="/var/lib/kubelet/pods/04478882-e34f-4b60-9a94-2a60fbfa7979/volumes"
Jul 23 09:40:53.912738 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:40:53.912725    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" path="/var/lib/kubelet/pods/f9d964b3-827e-4a96-b0d6-507021f6ef7b/volumes"
Jul 23 09:44:40.120009 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.119977    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120374    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04478882-e34f-4b60-9a94-2a60fbfa7979" containerName="main"
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120385    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="04478882-e34f-4b60-9a94-2a60fbfa7979" containerName="main"
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120402    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" containerName="main"
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120408    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" containerName="main"
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120472    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d964b3-827e-4a96-b0d6-507021f6ef7b" containerName="main"
Jul 23 09:44:40.122667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.120480    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="04478882-e34f-4b60-9a94-2a60fbfa7979" containerName="main"
Jul 23 09:44:40.123538 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.123518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.128040 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.128014    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0c364a76-serving-cert\""
Jul 23 09:44:40.128294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.128279    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:44:40.139470 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.139447    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:44:40.199123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.199089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.199123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.199123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mkl\" (UniqueName: \"kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.199329 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.199153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.299789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.299745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.299789 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.299790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mkl\" (UniqueName: \"kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.300096 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.299833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.300096 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:44:40.300015    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-serving-cert: secret "maas-api-e2e-ait-0c364a76-serving-cert" not found
Jul 23 09:44:40.300096 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:44:40.300087    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls podName:ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a nodeName:}" failed. No retries permitted until 2026-07-23 09:44:40.800064796 +0000 UTC m=+1665.478282562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls") pod "maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" (UID: "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a") : secret "maas-api-e2e-ait-0c364a76-serving-cert" not found
Jul 23 09:44:40.300415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.300392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.311883 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.311859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mkl\" (UniqueName: \"kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.804661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.804628    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:40.807060 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:40.807040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") pod \"maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:41.035732 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:41.035691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:41.175225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:41.175199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:44:41.177926 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:44:41.177884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee18dda9_d0ba_40dd_aa7f_ed1867bdb35a.slice/crio-eb1e0da1a9ac4f2ed7f5f591e87cd8524f53f17ff5d42999f45fe9601140cc4d WatchSource:0}: Error finding container eb1e0da1a9ac4f2ed7f5f591e87cd8524f53f17ff5d42999f45fe9601140cc4d: Status 404 returned error can't find the container with id eb1e0da1a9ac4f2ed7f5f591e87cd8524f53f17ff5d42999f45fe9601140cc4d
Jul 23 09:44:41.179549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:41.179516    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:44:41.762657 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:41.762622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" event={"ID":"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a","Type":"ContainerStarted","Data":"eb1e0da1a9ac4f2ed7f5f591e87cd8524f53f17ff5d42999f45fe9601140cc4d"}
Jul 23 09:44:42.768790 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:42.768752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" event={"ID":"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a","Type":"ContainerStarted","Data":"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"}
Jul 23 09:44:42.769280 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:42.768889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:44:42.788860 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:42.788816    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" podStartSLOduration=1.348478893 podStartE2EDuration="2.788804189s" totalImagesPullingTime="1.440325296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:44:42.787802516 +0000 UTC m=+1667.466020334" watchObservedRunningTime="2026-07-23 09:44:42.788804189 +0000 UTC m=+1667.467021971"
Jul 23 09:44:48.780558 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:44:48.780526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:45:00.139423 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.139385    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"]
Jul 23 09:45:00.143106 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.143083    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"]
Jul 23 09:45:00.143258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.143197    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"
Jul 23 09:45:00.146266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.146245    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\""
Jul 23 09:45:00.146399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.146327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"
Jul 23 09:45:00.149792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.149771    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"]
Jul 23 09:45:00.152436 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.152377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"]
Jul 23 09:45:00.158049 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.158025    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"
Jul 23 09:45:00.165391 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.165362    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"
Jul 23 09:45:00.320357 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.320326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"]
Jul 23 09:45:00.323264 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:45:00.323234    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4817ef32_8b00_48ff_895a_256e1deb53b1.slice/crio-da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c WatchSource:0}: Error finding container da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c: Status 404 returned error can't find the container with id da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c
Jul 23 09:45:00.345632 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.345608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"]
Jul 23 09:45:00.347282 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:45:00.347255    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865a3daf_2d2a_4559_bdb0_e57dbf559250.slice/crio-49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef WatchSource:0}: Error finding container 49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef: Status 404 returned error can't find the container with id 49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef
Jul 23 09:45:00.859272 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.859170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj" event={"ID":"4817ef32-8b00-48ff-895a-256e1deb53b1","Type":"ContainerStarted","Data":"26a6438533c18a0ba3b065f6086993a2a7c7978b67dc291d771b6a9e9cfe253c"}
Jul 23 09:45:00.859426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.859289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj" event={"ID":"4817ef32-8b00-48ff-895a-256e1deb53b1","Type":"ContainerStarted","Data":"da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c"}
Jul 23 09:45:00.860625 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.860594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r" event={"ID":"865a3daf-2d2a-4559-bdb0-e57dbf559250","Type":"ContainerStarted","Data":"e21046973bb3a57330994d8fc16af83883e3b00ce7d57528a84f201797e9dfeb"}
Jul 23 09:45:00.860756 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.860640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r" event={"ID":"865a3daf-2d2a-4559-bdb0-e57dbf559250","Type":"ContainerStarted","Data":"49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef"}
Jul 23 09:45:00.877187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.877138    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj" podStartSLOduration=0.877124216 podStartE2EDuration="877.124216ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:00.875462221 +0000 UTC m=+1685.553680031" watchObservedRunningTime="2026-07-23 09:45:00.877124216 +0000 UTC m=+1685.555342001"
Jul 23 09:45:00.891655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:00.891616    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r" podStartSLOduration=0.891603513 podStartE2EDuration="891.603513ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:00.890493591 +0000 UTC m=+1685.568711399" watchObservedRunningTime="2026-07-23 09:45:00.891603513 +0000 UTC m=+1685.569821335"
Jul 23 09:45:01.869224 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:01.869191    2560 generic.go:356] "Generic (PLEG): container finished" podID="4817ef32-8b00-48ff-895a-256e1deb53b1" containerID="26a6438533c18a0ba3b065f6086993a2a7c7978b67dc291d771b6a9e9cfe253c" exitCode=0
Jul 23 09:45:01.869669 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:01.869285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj" event={"ID":"4817ef32-8b00-48ff-895a-256e1deb53b1","Type":"ContainerDied","Data":"26a6438533c18a0ba3b065f6086993a2a7c7978b67dc291d771b6a9e9cfe253c"}
Jul 23 09:45:01.870807 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:01.870783    2560 generic.go:356] "Generic (PLEG): container finished" podID="865a3daf-2d2a-4559-bdb0-e57dbf559250" containerID="e21046973bb3a57330994d8fc16af83883e3b00ce7d57528a84f201797e9dfeb" exitCode=0
Jul 23 09:45:01.870943 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:01.870816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r" event={"ID":"865a3daf-2d2a-4559-bdb0-e57dbf559250","Type":"ContainerDied","Data":"e21046973bb3a57330994d8fc16af83883e3b00ce7d57528a84f201797e9dfeb"}
Jul 23 09:45:03.007067 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.007039    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"
Jul 23 09:45:03.041435 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.041403    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"
Jul 23 09:45:03.883086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.883061    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"
Jul 23 09:45:03.883322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.883104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj" event={"ID":"4817ef32-8b00-48ff-895a-256e1deb53b1","Type":"ContainerDied","Data":"da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c"}
Jul 23 09:45:03.883322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.883149    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da58be7a640a844ff70c29b2a9f77229ce3bbb6787f1f812b9ac74852b60d12c"
Jul 23 09:45:03.884387 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.884370    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r"
Jul 23 09:45:03.884497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.884386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746665-dgr7r" event={"ID":"865a3daf-2d2a-4559-bdb0-e57dbf559250","Type":"ContainerDied","Data":"49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef"}
Jul 23 09:45:03.884497 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:03.884416    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ed189d632441b528f5cf584ebb0cb44a7570528f6c34f7361af0ca996020ef"
Jul 23 09:45:05.822397 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822354    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"]
Jul 23 09:45:05.822881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822788    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4817ef32-8b00-48ff-895a-256e1deb53b1" containerName="cleanup"
Jul 23 09:45:05.822881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822801    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4817ef32-8b00-48ff-895a-256e1deb53b1" containerName="cleanup"
Jul 23 09:45:05.822881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822814    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="865a3daf-2d2a-4559-bdb0-e57dbf559250" containerName="cleanup"
Jul 23 09:45:05.822881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822819    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="865a3daf-2d2a-4559-bdb0-e57dbf559250" containerName="cleanup"
Jul 23 09:45:05.823106 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822898    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="865a3daf-2d2a-4559-bdb0-e57dbf559250" containerName="cleanup"
Jul 23 09:45:05.823106 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.822907    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4817ef32-8b00-48ff-895a-256e1deb53b1" containerName="cleanup"
Jul 23 09:45:05.826183 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.826159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:05.830004 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.829978    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n4hmw\""
Jul 23 09:45:05.836129 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.836094    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"]
Jul 23 09:45:05.931013 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:05.930978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c364a76-szljh\" (UID: \"42f34ee9-5615-4b94-9f5d-4b2c695051bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:06.032859 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.032425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c364a76-szljh\" (UID: \"42f34ee9-5615-4b94-9f5d-4b2c695051bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:06.033267 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.033239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0c364a76-szljh\" (UID: \"42f34ee9-5615-4b94-9f5d-4b2c695051bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:06.137261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.137179    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:06.483083 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.483059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"]
Jul 23 09:45:06.485647 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:45:06.485620    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f34ee9_5615_4b94_9f5d_4b2c695051bd.slice/crio-f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91 WatchSource:0}: Error finding container f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91: Status 404 returned error can't find the container with id f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91
Jul 23 09:45:06.905145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.905058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh" event={"ID":"42f34ee9-5615-4b94-9f5d-4b2c695051bd","Type":"ContainerStarted","Data":"24fdb1ddb4e52436f39d497b7ff48ac7f4a6c75f8d2035d636ef0afd47d66238"}
Jul 23 09:45:06.905145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.905118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh" event={"ID":"42f34ee9-5615-4b94-9f5d-4b2c695051bd","Type":"ContainerStarted","Data":"f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91"}
Jul 23 09:45:06.922869 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:06.922818    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh" podStartSLOduration=1.922803866 podStartE2EDuration="1.922803866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:06.920472478 +0000 UTC m=+1691.598690268" watchObservedRunningTime="2026-07-23 09:45:06.922803866 +0000 UTC m=+1691.601021650"
Jul 23 09:45:07.912334 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:07.912303    2560 generic.go:356] "Generic (PLEG): container finished" podID="42f34ee9-5615-4b94-9f5d-4b2c695051bd" containerID="24fdb1ddb4e52436f39d497b7ff48ac7f4a6c75f8d2035d636ef0afd47d66238" exitCode=0
Jul 23 09:45:07.912692 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:07.912503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh" event={"ID":"42f34ee9-5615-4b94-9f5d-4b2c695051bd","Type":"ContainerDied","Data":"24fdb1ddb4e52436f39d497b7ff48ac7f4a6c75f8d2035d636ef0afd47d66238"}
Jul 23 09:45:09.061150 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.061126    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:09.161337 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.161309    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca\") pod \"42f34ee9-5615-4b94-9f5d-4b2c695051bd\" (UID: \"42f34ee9-5615-4b94-9f5d-4b2c695051bd\") "
Jul 23 09:45:09.161686 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.161666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42f34ee9-5615-4b94-9f5d-4b2c695051bd" (UID: "42f34ee9-5615-4b94-9f5d-4b2c695051bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:45:09.262927 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.262837    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f34ee9-5615-4b94-9f5d-4b2c695051bd-maas-api-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:45:09.924409 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.924386    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"
Jul 23 09:45:09.924519 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.924386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh" event={"ID":"42f34ee9-5615-4b94-9f5d-4b2c695051bd","Type":"ContainerDied","Data":"f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91"}
Jul 23 09:45:09.924519 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:09.924504    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f43b0f0297e9c6bfbceb9c882288e3eb7cf8426b1b8d47de60f25ca8ca147f91"
Jul 23 09:45:16.240394 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.240362    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:45:16.240949 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.240867    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" podUID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" containerName="maas-api" containerID="cri-o://ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1" gracePeriod=30
Jul 23 09:45:16.290725 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.290688    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"]
Jul 23 09:45:16.296282 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.296253    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0c364a76-29746665-trzkj"]
Jul 23 09:45:16.490205 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.490179    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:45:16.626090 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.626008    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca\") pod \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") "
Jul 23 09:45:16.626090 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.626088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") pod \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") "
Jul 23 09:45:16.626325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.626161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94mkl\" (UniqueName: \"kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl\") pod \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\" (UID: \"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a\") "
Jul 23 09:45:16.626426 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.626400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" (UID: "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:45:16.628212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.628183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" (UID: "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:45:16.628301 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.628235    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl" (OuterVolumeSpecName: "kube-api-access-94mkl") pod "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" (UID: "ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a"). InnerVolumeSpecName "kube-api-access-94mkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:45:16.727169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.727134    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:45:16.727169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.727162    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mkl\" (UniqueName: \"kubernetes.io/projected/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-kube-api-access-94mkl\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:45:16.727169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.727172    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:45:16.959144 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.959113    2560 generic.go:356] "Generic (PLEG): container finished" podID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" containerID="ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1" exitCode=0
Jul 23 09:45:16.959327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.959158    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"
Jul 23 09:45:16.959327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.959200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" event={"ID":"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a","Type":"ContainerDied","Data":"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"}
Jul 23 09:45:16.959327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.959257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7" event={"ID":"ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a","Type":"ContainerDied","Data":"eb1e0da1a9ac4f2ed7f5f591e87cd8524f53f17ff5d42999f45fe9601140cc4d"}
Jul 23 09:45:16.959327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.959285    2560 scope.go:117] "RemoveContainer" containerID="ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"
Jul 23 09:45:16.968306 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.968289    2560 scope.go:117] "RemoveContainer" containerID="ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"
Jul 23 09:45:16.968549 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:45:16.968533    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1\": container with ID starting with ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1 not found: ID does not exist" containerID="ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"
Jul 23 09:45:16.968600 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.968558    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1"} err="failed to get container status \"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1\": rpc error: code = NotFound desc = could not find container \"ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1\": container with ID starting with ed1195fde0359c0fe0cb460e74d9473f40e3fd644d1aea5263129d518a48ccd1 not found: ID does not exist"
Jul 23 09:45:16.981338 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.981313    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:45:16.986090 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:16.986068    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0c364a76-8497b8698c-lqrl7"]
Jul 23 09:45:17.912122 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:17.912053    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4817ef32-8b00-48ff-895a-256e1deb53b1" path="/var/lib/kubelet/pods/4817ef32-8b00-48ff-895a-256e1deb53b1/volumes"
Jul 23 09:45:17.912474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:17.912457    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" path="/var/lib/kubelet/pods/ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a/volumes"
Jul 23 09:45:20.900979 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:20.900944    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"]
Jul 23 09:45:20.902683 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:20.902663    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0c364a76-szljh"]
Jul 23 09:45:21.912941 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:21.912884    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f34ee9-5615-4b94-9f5d-4b2c695051bd" path="/var/lib/kubelet/pods/42f34ee9-5615-4b94-9f5d-4b2c695051bd/volumes"
Jul 23 09:45:22.180011 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.179930    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:45:22.180336 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180322    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" containerName="maas-api"
Jul 23 09:45:22.180378 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180337    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" containerName="maas-api"
Jul 23 09:45:22.180378 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180354    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f34ee9-5615-4b94-9f5d-4b2c695051bd" containerName="revoke-keys"
Jul 23 09:45:22.180378 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180360    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f34ee9-5615-4b94-9f5d-4b2c695051bd" containerName="revoke-keys"
Jul 23 09:45:22.180485 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180419    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee18dda9-d0ba-40dd-aa7f-ed1867bdb35a" containerName="maas-api"
Jul 23 09:45:22.180485 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.180430    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f34ee9-5615-4b94-9f5d-4b2c695051bd" containerName="revoke-keys"
Jul 23 09:45:22.185121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.185100    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.187738 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.187716    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:45:22.187873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.187854    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bd6e3a7b-gw-openshift-default-dockercfg-lwxzk\""
Jul 23 09:45:22.199458 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.199434    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:45:22.279361 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.279751 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.279665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ld7\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380201 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380201 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ld7\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380435 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380435 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380435 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380435 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380477    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380862 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380862 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.380965 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.380907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.381362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.381294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.383027 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.383000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.383367 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.383346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.421374 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.421344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ld7\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.421501 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.421351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token\") pod \"e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") " pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.495316 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.495235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:22.631263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.631233    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:45:22.633012 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:45:22.632982    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7a46c1_f7dc_4c57_9b9a_74c262d13968.slice/crio-4e7fbd307e84c2bd196a24ba2ed1a17df8752edfd3022148a4032213c88c8794 WatchSource:0}: Error finding container 4e7fbd307e84c2bd196a24ba2ed1a17df8752edfd3022148a4032213c88c8794: Status 404 returned error can't find the container with id 4e7fbd307e84c2bd196a24ba2ed1a17df8752edfd3022148a4032213c88c8794
Jul 23 09:45:22.991977 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:22.991943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerStarted","Data":"4e7fbd307e84c2bd196a24ba2ed1a17df8752edfd3022148a4032213c88c8794"}
Jul 23 09:45:26.049127 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:26.049087    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:26.049415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:26.049166    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:26.049415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:26.049197    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:27.021301 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:27.021262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerStarted","Data":"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"}
Jul 23 09:45:27.055164 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:27.055106    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podStartSLOduration=1.64145109 podStartE2EDuration="5.05509061s" totalImagesPullingTime="3.41363952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:27.05131644 +0000 UTC m=+1711.729534225" watchObservedRunningTime="2026-07-23 09:45:27.05509061 +0000 UTC m=+1711.733308455"
Jul 23 09:45:27.495636 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:27.495594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:27.500643 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:27.500613    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:28.025781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.025750    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:28.026578 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.026561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:28.814326 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.814296    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:45:28.817864 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.817842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.821518 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.821490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bd6e3a7b-serving-cert\""
Jul 23 09:45:28.821838 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.821822    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:45:28.839871 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.839851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.839983 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.839899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfj9\" (UniqueName: \"kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.840028 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.839997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.844596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.844573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:45:28.941077 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.941043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.941213 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.941147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.941279 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.941217    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfj9\" (UniqueName: \"kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.941332 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:45:28.941284    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-serving-cert: secret "maas-api-e2e-migrate-bd6e3a7b-serving-cert" not found
Jul 23 09:45:28.941388 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:45:28.941359    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls podName:9c4740bf-11f4-432e-acb7-02bc03f15627 nodeName:}" failed. No retries permitted until 2026-07-23 09:45:29.441328806 +0000 UTC m=+1714.119546583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls") pod "maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" (UID: "9c4740bf-11f4-432e-acb7-02bc03f15627") : secret "maas-api-e2e-migrate-bd6e3a7b-serving-cert" not found
Jul 23 09:45:28.941665 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.941642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:28.959298 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:28.959274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfj9\" (UniqueName: \"kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:29.446144 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:29.446102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:29.448732 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:29.448706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") pod \"maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:29.728196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:29.728110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:29.895910 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:29.895883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:45:29.897695 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:45:29.897664    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4740bf_11f4_432e_acb7_02bc03f15627.slice/crio-1f0b08d550ba8a7ca1a8e28660016c3e5220d16e9aec984df8d72d3f52576bc8 WatchSource:0}: Error finding container 1f0b08d550ba8a7ca1a8e28660016c3e5220d16e9aec984df8d72d3f52576bc8: Status 404 returned error can't find the container with id 1f0b08d550ba8a7ca1a8e28660016c3e5220d16e9aec984df8d72d3f52576bc8
Jul 23 09:45:30.037166 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:30.037066    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" event={"ID":"9c4740bf-11f4-432e-acb7-02bc03f15627","Type":"ContainerStarted","Data":"1f0b08d550ba8a7ca1a8e28660016c3e5220d16e9aec984df8d72d3f52576bc8"}
Jul 23 09:45:31.044157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:31.044129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" event={"ID":"9c4740bf-11f4-432e-acb7-02bc03f15627","Type":"ContainerStarted","Data":"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"}
Jul 23 09:45:31.044594 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:31.044252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:31.064571 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:31.064526    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" podStartSLOduration=2.021316132 podStartE2EDuration="3.064511048s" totalImagesPullingTime="1.043194916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:45:31.063001498 +0000 UTC m=+1715.741219304" watchObservedRunningTime="2026-07-23 09:45:31.064511048 +0000 UTC m=+1715.742728830"
Jul 23 09:45:37.059014 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:37.058981    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:45:43.027758 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.027673    2560 patch_prober.go:28] interesting pod/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 23 09:45:43.027758 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.027728    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 23 09:45:43.107145 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.107114    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerID="5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828" exitCode=0
Jul 23 09:45:43.107306 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.107183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerDied","Data":"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"}
Jul 23 09:45:43.107582 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.107568    2560 scope.go:117] "RemoveContainer" containerID="5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"
Jul 23 09:45:43.108529 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.108498    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:43.108609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.108565    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:43.108609 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:43.108598    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:45:44.113757 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:44.113728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerStarted","Data":"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"}
Jul 23 09:45:44.495611 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:44.495575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:44.495806 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:44.495784    2560 patch_prober.go:28] interesting pod/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 23 09:45:44.495851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:44.495831    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 23 09:45:45.496592 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:45.496554    2560 patch_prober.go:28] interesting pod/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 23 09:45:45.496990 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:45.496612    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 23 09:45:46.495892 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:46.495856    2560 patch_prober.go:28] interesting pod/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 23 09:45:46.496158 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:46.495933    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 23 09:45:47.496192 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:47.496154    2560 patch_prober.go:28] interesting pod/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 23 09:45:47.496568 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:47.496213    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 23 09:45:48.499823 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:48.499794    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:48.500229 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:48.500100    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:48.500855 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:48.500839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:45:55.938870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:45:55.938827    2560 scope.go:117] "RemoveContainer" containerID="0eeab33630f479926a29297c3f5a61220e0992141d1d4f0263fb2020e70829b5"
Jul 23 09:46:04.304771 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.304724    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:46:04.305286 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.305121    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" podUID="9c4740bf-11f4-432e-acb7-02bc03f15627" containerName="maas-api" containerID="cri-o://b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c" gracePeriod=30
Jul 23 09:46:04.568135 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.568074    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:46:04.577954 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.577932    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca\") pod \"9c4740bf-11f4-432e-acb7-02bc03f15627\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") "
Jul 23 09:46:04.578039 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.577969    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") pod \"9c4740bf-11f4-432e-acb7-02bc03f15627\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") "
Jul 23 09:46:04.578099 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.578044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgfj9\" (UniqueName: \"kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9\") pod \"9c4740bf-11f4-432e-acb7-02bc03f15627\" (UID: \"9c4740bf-11f4-432e-acb7-02bc03f15627\") "
Jul 23 09:46:04.578237 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.578213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c4740bf-11f4-432e-acb7-02bc03f15627" (UID: "9c4740bf-11f4-432e-acb7-02bc03f15627"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:46:04.578322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.578311    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c4740bf-11f4-432e-acb7-02bc03f15627-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:04.579963 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.579943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c4740bf-11f4-432e-acb7-02bc03f15627" (UID: "9c4740bf-11f4-432e-acb7-02bc03f15627"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:46:04.580056 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.580038    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9" (OuterVolumeSpecName: "kube-api-access-wgfj9") pod "9c4740bf-11f4-432e-acb7-02bc03f15627" (UID: "9c4740bf-11f4-432e-acb7-02bc03f15627"). InnerVolumeSpecName "kube-api-access-wgfj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:46:04.679341 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.679305    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgfj9\" (UniqueName: \"kubernetes.io/projected/9c4740bf-11f4-432e-acb7-02bc03f15627-kube-api-access-wgfj9\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:04.679341 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:04.679335    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c4740bf-11f4-432e-acb7-02bc03f15627-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:05.219531 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.219494    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c4740bf-11f4-432e-acb7-02bc03f15627" containerID="b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c" exitCode=0
Jul 23 09:46:05.219531 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.219535    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"
Jul 23 09:46:05.219774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.219576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" event={"ID":"9c4740bf-11f4-432e-acb7-02bc03f15627","Type":"ContainerDied","Data":"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"}
Jul 23 09:46:05.219774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.219627    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp" event={"ID":"9c4740bf-11f4-432e-acb7-02bc03f15627","Type":"ContainerDied","Data":"1f0b08d550ba8a7ca1a8e28660016c3e5220d16e9aec984df8d72d3f52576bc8"}
Jul 23 09:46:05.219774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.219649    2560 scope.go:117] "RemoveContainer" containerID="b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"
Jul 23 09:46:05.229166 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.229151    2560 scope.go:117] "RemoveContainer" containerID="b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"
Jul 23 09:46:05.229371 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:46:05.229353    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c\": container with ID starting with b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c not found: ID does not exist" containerID="b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"
Jul 23 09:46:05.229423 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.229378    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c"} err="failed to get container status \"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c\": rpc error: code = NotFound desc = could not find container \"b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c\": container with ID starting with b460f82c7066d9ef663da1dbb0a72fd714e4dc973d3cf4e8dbe56f6c00d99d5c not found: ID does not exist"
Jul 23 09:46:05.243119 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.243094    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:46:05.247874 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.247855    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bd6e3a7b-6b65f84b-54lgp"]
Jul 23 09:46:05.914080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:05.914046    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4740bf-11f4-432e-acb7-02bc03f15627" path="/var/lib/kubelet/pods/9c4740bf-11f4-432e-acb7-02bc03f15627/volumes"
Jul 23 09:46:08.998149 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:08.998112    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:46:08.998543 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:08.998373    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy" containerID="cri-o://119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f" gracePeriod=30
Jul 23 09:46:09.828903 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.828869    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:46:09.829335 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.829321    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4740bf-11f4-432e-acb7-02bc03f15627" containerName="maas-api"
Jul 23 09:46:09.829410 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.829337    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4740bf-11f4-432e-acb7-02bc03f15627" containerName="maas-api"
Jul 23 09:46:09.829410 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.829402    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4740bf-11f4-432e-acb7-02bc03f15627" containerName="maas-api"
Jul 23 09:46:09.834071 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.834049    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.836930 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.836891    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ee6db4d-openshift-default-dockercfg-zkhxs\""
Jul 23 09:46:09.847375 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.847346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:46:09.926802 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.926764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927081 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927246 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927246 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927246 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927246 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927246 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcmn\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927658 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:09.927658 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:09.927362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.028763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.028763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.028969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcmn\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029289 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029373    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.029731 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.029650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.031433 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.031411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.031781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.031757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.036963 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.036943    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.037485 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.037454    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcmn\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn\") pod \"e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") " pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.147904 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.147815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:10.291501 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.291478    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:46:10.293681 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:46:10.293655    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a2aaaf_894a_4548_83df_fd43f96a5c67.slice/crio-95beeb414c88cda8bc9ac857c05beb2f183adbdaa2b7194ce7a929d198cef551 WatchSource:0}: Error finding container 95beeb414c88cda8bc9ac857c05beb2f183adbdaa2b7194ce7a929d198cef551: Status 404 returned error can't find the container with id 95beeb414c88cda8bc9ac857c05beb2f183adbdaa2b7194ce7a929d198cef551
Jul 23 09:46:10.295724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.295682    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:10.295808 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.295767    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:10.295808 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:10.295796    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:11.255030 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:11.254988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerStarted","Data":"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"}
Jul 23 09:46:11.255440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:11.255047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerStarted","Data":"95beeb414c88cda8bc9ac857c05beb2f183adbdaa2b7194ce7a929d198cef551"}
Jul 23 09:46:11.277970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:11.277890    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podStartSLOduration=2.277876441 podStartE2EDuration="2.277876441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:46:11.277208956 +0000 UTC m=+1755.955426766" watchObservedRunningTime="2026-07-23 09:46:11.277876441 +0000 UTC m=+1755.956094226"
Jul 23 09:46:12.148185 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:12.148152    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:12.153258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:12.153229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:12.260778 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:12.260745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:12.261819 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:12.261799    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:14.252332 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.252308    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:46:14.272597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.272530    2560 generic.go:356] "Generic (PLEG): container finished" podID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerID="119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f" exitCode=0
Jul 23 09:46:14.272597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.272571    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"
Jul 23 09:46:14.272763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.272619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerDied","Data":"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"}
Jul 23 09:46:14.272763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.272668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg" event={"ID":"5e7a46c1-f7dc-4c57-9b9a-74c262d13968","Type":"ContainerDied","Data":"4e7fbd307e84c2bd196a24ba2ed1a17df8752edfd3022148a4032213c88c8794"}
Jul 23 09:46:14.272763 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.272695    2560 scope.go:117] "RemoveContainer" containerID="119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"
Jul 23 09:46:14.287860 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.285880    2560 scope.go:117] "RemoveContainer" containerID="5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"
Jul 23 09:46:14.296749 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.296729    2560 scope.go:117] "RemoveContainer" containerID="119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"
Jul 23 09:46:14.297085 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:46:14.297066    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f\": container with ID starting with 119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f not found: ID does not exist" containerID="119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"
Jul 23 09:46:14.297142 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.297093    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f"} err="failed to get container status \"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f\": rpc error: code = NotFound desc = could not find container \"119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f\": container with ID starting with 119d1990d6b7b7973505bc3d7b348295840482fb499f49a35ffc46c08cfc7d4f not found: ID does not exist"
Jul 23 09:46:14.297142 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.297110    2560 scope.go:117] "RemoveContainer" containerID="5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"
Jul 23 09:46:14.297387 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:46:14.297365    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828\": container with ID starting with 5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828 not found: ID does not exist" containerID="5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"
Jul 23 09:46:14.297441 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.297394    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828"} err="failed to get container status \"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828\": rpc error: code = NotFound desc = could not find container \"5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828\": container with ID starting with 5cf854a3f164c452bc107711b5030f937af08c492fe35efba04d3b51761de828 not found: ID does not exist"
Jul 23 09:46:14.368825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.368792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.368825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.368827    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67ld7\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.368859    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.368882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369086 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.368996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369115    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs\") pod \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\" (UID: \"5e7a46c1-f7dc-4c57-9b9a-74c262d13968\") "
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:46:14.369704 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:46:14.370147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369821    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.370147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369841    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.370147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369856    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.370147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.369988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:46:14.370589 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.370345    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:46:14.371445 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.371421    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7" (OuterVolumeSpecName: "kube-api-access-67ld7") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "kube-api-access-67ld7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:46:14.371537 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.371448    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:46:14.371807 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.371792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:46:14.371974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.371957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e7a46c1-f7dc-4c57-9b9a-74c262d13968" (UID: "5e7a46c1-f7dc-4c57-9b9a-74c262d13968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:46:14.471194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471162    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.471194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471189    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ld7\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-kube-api-access-67ld7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.471194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471198    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.471416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471207    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.471416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471214    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.471416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.471222    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7a46c1-f7dc-4c57-9b9a-74c262d13968-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:46:14.597450 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.597414    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:46:14.603287 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:14.603261    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bd6e3a7b-gw-openshift-default-56c7cdb97f-r4dvg"]
Jul 23 09:46:15.914184 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:15.914150    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" path="/var/lib/kubelet/pods/5e7a46c1-f7dc-4c57-9b9a-74c262d13968/volumes"
Jul 23 09:46:28.049892 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.049855    2560 patch_prober.go:28] interesting pod/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:32996->10.133.0.55:15021: read: connection reset by peer" start-of-body=
Jul 23 09:46:28.050266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.049948    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:32996->10.133.0.55:15021: read: connection reset by peer"
Jul 23 09:46:28.349405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.349307    2560 generic.go:356] "Generic (PLEG): container finished" podID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerID="9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558" exitCode=0
Jul 23 09:46:28.349405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.349359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerDied","Data":"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"}
Jul 23 09:46:28.349745 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.349730    2560 scope.go:117] "RemoveContainer" containerID="9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"
Jul 23 09:46:28.350651 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.350622    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:28.350757 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.350688    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:28.350757 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:28.350729    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:29.358964 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:29.358903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerStarted","Data":"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"}
Jul 23 09:46:30.148883 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:30.148848    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:30.149296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:30.149191    2560 patch_prober.go:28] interesting pod/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 23 09:46:30.149459 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:30.149435    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 23 09:46:31.148931 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:31.148862    2560 patch_prober.go:28] interesting pod/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 23 09:46:31.149440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:31.148958    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 23 09:46:32.148438 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:32.148397    2560 patch_prober.go:28] interesting pod/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 23 09:46:32.148635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:32.148473    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 23 09:46:33.148365 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:33.148323    2560 patch_prober.go:28] interesting pod/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 23 09:46:33.148999 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:33.148956    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 23 09:46:34.153631 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:34.153604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:34.154044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:34.153860    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:34.154512 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:34.154495    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:46:54.969673 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:54.969635    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:46:54.970194 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:54.969886    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy" containerID="cri-o://72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8" gracePeriod=30
Jul 23 09:46:55.791781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.791752    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:46:55.792269 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.792253    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:46:55.792328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.792272    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:46:55.792366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.792353    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:46:55.792366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.792363    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:46:55.794333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.794319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.796865 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.796841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0b80ea18-openshift-default-dockercfg-2m82f\""
Jul 23 09:46:55.807533 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.807508    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:46:55.930680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gv2\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.930950 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.931229 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:55.931229 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:55.930996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032226 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032283    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032320    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gv2\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.032655 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.033109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.033109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.033109 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.032856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.033350 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.033323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.033509 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.033482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.034664 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.034644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.035454 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.035432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.040524 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.040498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.040842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.040823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gv2\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2\") pod \"e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") " pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.104968 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.104898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:56.245094 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.245068    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:46:56.246993 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:46:56.246964    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602d28f3_4f52_4fc3_b240_065e8f312bc4.slice/crio-cbddac2f21531d020777f555c8848c3b259a38151a4694435eaaf12022a9c573 WatchSource:0}: Error finding container cbddac2f21531d020777f555c8848c3b259a38151a4694435eaaf12022a9c573: Status 404 returned error can't find the container with id cbddac2f21531d020777f555c8848c3b259a38151a4694435eaaf12022a9c573
Jul 23 09:46:56.250717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.250673    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:56.250824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.250768    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:56.250824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.250806    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:46:56.498658 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.498617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerStarted","Data":"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"}
Jul 23 09:46:56.498859 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.498673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerStarted","Data":"cbddac2f21531d020777f555c8848c3b259a38151a4694435eaaf12022a9c573"}
Jul 23 09:46:56.521740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:56.521680    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podStartSLOduration=1.521661662 podStartE2EDuration="1.521661662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:46:56.518808771 +0000 UTC m=+1801.197026577" watchObservedRunningTime="2026-07-23 09:46:56.521661662 +0000 UTC m=+1801.199879446"
Jul 23 09:46:57.105792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:57.105762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:57.107074 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:57.107052    2560 patch_prober.go:28] interesting pod/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 23 09:46:57.107185 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:57.107102    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 23 09:46:58.109612 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:58.109586    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:58.509740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:58.509711    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:46:58.510974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:46:58.510945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:00.240334 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.240258    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:47:00.376599 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376565    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376613    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376650    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376680    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376728    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376752    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.376787 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.377140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376818    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbcmn\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.377140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.376904    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs\") pod \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\" (UID: \"f4a2aaaf-894a-4548-83df-fd43f96a5c67\") "
Jul 23 09:47:00.377140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:00.377140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:00.377140 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377113    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:47:00.377401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377262    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.377401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377283    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.377401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377299    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.377569 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:00.377680 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.377650    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:00.379221 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.379199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:47:00.379357 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.379223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn" (OuterVolumeSpecName: "kube-api-access-xbcmn") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "kube-api-access-xbcmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:47:00.379357 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.379325    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:47:00.379496 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.379476    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4a2aaaf-894a-4548-83df-fd43f96a5c67" (UID: "f4a2aaaf-894a-4548-83df-fd43f96a5c67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:00.477788 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477757    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.477788 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477782    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.477788 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477790    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.478024 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477799    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.478024 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477809    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a2aaaf-894a-4548-83df-fd43f96a5c67-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.478024 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.477821    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbcmn\" (UniqueName: \"kubernetes.io/projected/f4a2aaaf-894a-4548-83df-fd43f96a5c67-kube-api-access-xbcmn\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:00.523605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.523527    2560 generic.go:356] "Generic (PLEG): container finished" podID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerID="72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8" exitCode=0
Jul 23 09:47:00.523605 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.523578    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"
Jul 23 09:47:00.523769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.523605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerDied","Data":"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"}
Jul 23 09:47:00.523769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.523681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq" event={"ID":"f4a2aaaf-894a-4548-83df-fd43f96a5c67","Type":"ContainerDied","Data":"95beeb414c88cda8bc9ac857c05beb2f183adbdaa2b7194ce7a929d198cef551"}
Jul 23 09:47:00.523769 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.523707    2560 scope.go:117] "RemoveContainer" containerID="72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"
Jul 23 09:47:00.535327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.535297    2560 scope.go:117] "RemoveContainer" containerID="9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"
Jul 23 09:47:00.544230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.544210    2560 scope.go:117] "RemoveContainer" containerID="72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"
Jul 23 09:47:00.544700 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:47:00.544568    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8\": container with ID starting with 72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8 not found: ID does not exist" containerID="72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"
Jul 23 09:47:00.544700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.544632    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8"} err="failed to get container status \"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8\": rpc error: code = NotFound desc = could not find container \"72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8\": container with ID starting with 72d77f5ba378706ccb41e307d06042686ee1a22268b084a974442e3614b6dea8 not found: ID does not exist"
Jul 23 09:47:00.544700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.544654    2560 scope.go:117] "RemoveContainer" containerID="9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"
Jul 23 09:47:00.544975 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:47:00.544946    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558\": container with ID starting with 9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558 not found: ID does not exist" containerID="9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"
Jul 23 09:47:00.545105 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.544980    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558"} err="failed to get container status \"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558\": rpc error: code = NotFound desc = could not find container \"9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558\": container with ID starting with 9430bea8ec54a99643bf277f0ef09042a8d473a43b8b70ea357b74a33cadb558 not found: ID does not exist"
Jul 23 09:47:00.547241 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.547222    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:47:00.551304 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:00.551282    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ee6db4d-openshift-default-6f84765df5-fbrbq"]
Jul 23 09:47:01.912909 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:01.912874    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" path="/var/lib/kubelet/pods/f4a2aaaf-894a-4548-83df-fd43f96a5c67/volumes"
Jul 23 09:47:12.597664 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.597578    2560 generic.go:356] "Generic (PLEG): container finished" podID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerID="26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d" exitCode=0
Jul 23 09:47:12.597664 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.597632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerDied","Data":"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"}
Jul 23 09:47:12.598112 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.598033    2560 scope.go:117] "RemoveContainer" containerID="26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"
Jul 23 09:47:12.599010 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.598979    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:47:12.599125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.599045    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:47:12.599125 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:12.599076    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:47:13.510635 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:13.510591    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:13.604408 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:13.604375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerStarted","Data":"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"}
Jul 23 09:47:14.105967 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:14.105932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:14.107219 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:14.107198    2560 patch_prober.go:28] interesting pod/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 23 09:47:14.107347 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:14.107239    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 23 09:47:15.106318 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:15.106290    2560 patch_prober.go:28] interesting pod/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 23 09:47:15.106684 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:15.106340    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 23 09:47:16.105549 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:16.105508    2560 patch_prober.go:28] interesting pod/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 23 09:47:16.105734 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:16.105563    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 23 09:47:17.105372 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:17.105342    2560 patch_prober.go:28] interesting pod/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 23 09:47:17.105739 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:17.105395    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 23 09:47:18.109676 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:18.109649    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:18.110078 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:18.109933    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:18.110560 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:18.110543    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:38.035134 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:38.035093    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:47:38.035743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:38.035361    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy" containerID="cri-o://beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4" gracePeriod=30
Jul 23 09:47:43.190878 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.190855    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:43.359717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.359717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359671    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.359717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359697    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359744    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5gv2\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359830    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360022 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.359967    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo\") pod \"602d28f3-4f52-4fc3-b240-065e8f312bc4\" (UID: \"602d28f3-4f52-4fc3-b240-065e8f312bc4\") "
Jul 23 09:47:43.360328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:43.360328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:43.360328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360200    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:47:43.360328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360303    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/602d28f3-4f52-4fc3-b240-065e8f312bc4-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.360328 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360323    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.360565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360336    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.360565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360393    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:43.360784 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.360760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:43.362348 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.362324    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:47:43.362444 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.362409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:47:43.362508 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.362458    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:47:43.362508 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.362465    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2" (OuterVolumeSpecName: "kube-api-access-v5gv2") pod "602d28f3-4f52-4fc3-b240-065e8f312bc4" (UID: "602d28f3-4f52-4fc3-b240-065e8f312bc4"). InnerVolumeSpecName "kube-api-access-v5gv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:47:43.460715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460690    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.460715 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460712    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.460851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460722    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.460851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460730    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.460851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460738    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/602d28f3-4f52-4fc3-b240-065e8f312bc4-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.460851 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.460757    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gv2\" (UniqueName: \"kubernetes.io/projected/602d28f3-4f52-4fc3-b240-065e8f312bc4-kube-api-access-v5gv2\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:47:43.763136 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.763106    2560 generic.go:356] "Generic (PLEG): container finished" podID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerID="beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4" exitCode=0
Jul 23 09:47:43.763294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.763151    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"
Jul 23 09:47:43.763294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.763153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerDied","Data":"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"}
Jul 23 09:47:43.763391 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.763293    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd" event={"ID":"602d28f3-4f52-4fc3-b240-065e8f312bc4","Type":"ContainerDied","Data":"cbddac2f21531d020777f555c8848c3b259a38151a4694435eaaf12022a9c573"}
Jul 23 09:47:43.763391 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.763322    2560 scope.go:117] "RemoveContainer" containerID="beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"
Jul 23 09:47:43.779629 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.779611    2560 scope.go:117] "RemoveContainer" containerID="26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"
Jul 23 09:47:43.788102 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.788085    2560 scope.go:117] "RemoveContainer" containerID="beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"
Jul 23 09:47:43.788361 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:47:43.788339    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4\": container with ID starting with beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4 not found: ID does not exist" containerID="beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"
Jul 23 09:47:43.788439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.788372    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4"} err="failed to get container status \"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4\": rpc error: code = NotFound desc = could not find container \"beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4\": container with ID starting with beb665dbe17f61722d390b99d1d37a52843a3d2f420a9b92403a6aa2b6e73fc4 not found: ID does not exist"
Jul 23 09:47:43.788439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.788394    2560 scope.go:117] "RemoveContainer" containerID="26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"
Jul 23 09:47:43.788639 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:47:43.788622    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d\": container with ID starting with 26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d not found: ID does not exist" containerID="26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"
Jul 23 09:47:43.788681 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.788645    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d"} err="failed to get container status \"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d\": rpc error: code = NotFound desc = could not find container \"26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d\": container with ID starting with 26fd1a951bd9d63abdf1fd1b75858365d217a1cb4de23049a111f9aca8aac23d not found: ID does not exist"
Jul 23 09:47:43.796778 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.796758    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:47:43.801868 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.801847    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0b80ea18-openshift-default-575655b87-jdkpd"]
Jul 23 09:47:43.912544 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:47:43.912517    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" path="/var/lib/kubelet/pods/602d28f3-4f52-4fc3-b240-065e8f312bc4/volumes"
Jul 23 09:49:15.410505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410936    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410951    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e7a46c1-f7dc-4c57-9b9a-74c262d13968" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410967    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410972    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410980    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.410985    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411001    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411009    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411024    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411029    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411101    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411110    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="602d28f3-4f52-4fc3-b240-065e8f312bc4" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411118    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.411300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.411123    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a2aaaf-894a-4548-83df-fd43f96a5c67" containerName="istio-proxy"
Jul 23 09:49:15.414440 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.414424    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.417055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.417035    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18ba59c9-openshift-default-dockercfg-6vj2g\""
Jul 23 09:49:15.417213 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.417037    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:49:15.439072 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.439040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:15.560291 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560291 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s6s\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560517 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560766 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.560766 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.560642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.661868 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s6s\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.661868 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.661868 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662232 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.661980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662448 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662611 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.662611 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.663000 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.662972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.663100 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.663010    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.664829 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.664798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.665029 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.665013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.672488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.672462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s6s\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.673104 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.673082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token\") pod \"e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") " pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.724752 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.724719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:15.871072 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.871048    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:15.873755 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:49:15.873728    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c94fbe_ee64_4978_9029_b8ab4432f55d.slice/crio-6b9e6035cafad675b9c7d39befa2ff90fcadcdcba722e63e5f843ccbd0014c9e WatchSource:0}: Error finding container 6b9e6035cafad675b9c7d39befa2ff90fcadcdcba722e63e5f843ccbd0014c9e: Status 404 returned error can't find the container with id 6b9e6035cafad675b9c7d39befa2ff90fcadcdcba722e63e5f843ccbd0014c9e
Jul 23 09:49:15.876276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.876241    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:15.876405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.876317    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:15.876405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:15.876365    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:16.222487 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:16.222404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerStarted","Data":"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"}
Jul 23 09:49:16.222487 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:16.222457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerStarted","Data":"6b9e6035cafad675b9c7d39befa2ff90fcadcdcba722e63e5f843ccbd0014c9e"}
Jul 23 09:49:16.244572 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:16.244523    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podStartSLOduration=1.244507364 podStartE2EDuration="1.244507364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:49:16.242593782 +0000 UTC m=+1940.920811598" watchObservedRunningTime="2026-07-23 09:49:16.244507364 +0000 UTC m=+1940.922725146"
Jul 23 09:49:16.725299 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:16.725268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:16.730261 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:16.730226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:17.226986 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:17.226951    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:17.228080 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:17.228049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:18.603842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.603798    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:18.608187 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.608166    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.610702 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.610681    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-18ba59c9-serving-cert\""
Jul 23 09:49:18.610845 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.610828    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:49:18.616718 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.616692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:18.693034 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.693000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjnz\" (UniqueName: \"kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.693224 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.693045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.693224 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.693123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.794084 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.794049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.794250 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.794110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjnz\" (UniqueName: \"kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.794250 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.794152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.794250 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:49:18.794207    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-serving-cert: secret "maas-api-e2e-mt-18ba59c9-serving-cert" not found
Jul 23 09:49:18.794359 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:49:18.794274    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls podName:47afdb26-6303-42e4-a8f4-55e7c97d1f5a nodeName:}" failed. No retries permitted until 2026-07-23 09:49:19.294253808 +0000 UTC m=+1943.972471590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls") pod "maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" (UID: "47afdb26-6303-42e4-a8f4-55e7c97d1f5a") : secret "maas-api-e2e-mt-18ba59c9-serving-cert" not found
Jul 23 09:49:18.794842 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.794821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:18.806105 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:18.806075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjnz\" (UniqueName: \"kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:19.298788 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:19.298750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:19.301118 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:19.301098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") pod \"maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:19.521077 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:19.521037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:19.864991 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:19.864967    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:19.866874 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:49:19.866844    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47afdb26_6303_42e4_a8f4_55e7c97d1f5a.slice/crio-531116ba783185fdb7313cef60234a7fdcbb9f252291e6851e0346535a82ca54 WatchSource:0}: Error finding container 531116ba783185fdb7313cef60234a7fdcbb9f252291e6851e0346535a82ca54: Status 404 returned error can't find the container with id 531116ba783185fdb7313cef60234a7fdcbb9f252291e6851e0346535a82ca54
Jul 23 09:49:20.244157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:20.244120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" event={"ID":"47afdb26-6303-42e4-a8f4-55e7c97d1f5a","Type":"ContainerStarted","Data":"531116ba783185fdb7313cef60234a7fdcbb9f252291e6851e0346535a82ca54"}
Jul 23 09:49:21.253372 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:21.253336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" event={"ID":"47afdb26-6303-42e4-a8f4-55e7c97d1f5a","Type":"ContainerStarted","Data":"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"}
Jul 23 09:49:21.253864 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:21.253407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:21.272147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:21.272100    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" podStartSLOduration=2.836923071 podStartE2EDuration="3.272085088s" totalImagesPullingTime="435.162017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:49:21.270281299 +0000 UTC m=+1945.948499083" watchObservedRunningTime="2026-07-23 09:49:21.272085088 +0000 UTC m=+1945.950302872"
Jul 23 09:49:27.264894 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:27.264854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:29.296375 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.296343    2560 generic.go:356] "Generic (PLEG): container finished" podID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerID="5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1" exitCode=0
Jul 23 09:49:29.296781 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.296425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerDied","Data":"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"}
Jul 23 09:49:29.296871 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.296851    2560 scope.go:117] "RemoveContainer" containerID="5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"
Jul 23 09:49:29.297990 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.297959    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:29.298116 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.298029    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:29.298116 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:29.298064    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:49:30.303087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:30.303047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerStarted","Data":"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"}
Jul 23 09:49:30.725714 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:30.725680    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:30.726683 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:30.726661    2560 patch_prober.go:28] interesting pod/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 23 09:49:30.726754 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:30.726720    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 23 09:49:31.725321 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:31.725283    2560 patch_prober.go:28] interesting pod/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 23 09:49:31.725714 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:31.725344    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 23 09:49:32.227008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:32.226976    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:32.725903 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:32.725869    2560 patch_prober.go:28] interesting pod/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 23 09:49:32.726310 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:32.725937    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 23 09:49:33.726048 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:33.726009    2560 patch_prober.go:28] interesting pod/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Jul 23 09:49:33.726433 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:33.726071    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Jul 23 09:49:34.729672 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:34.729641    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:34.730759 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:34.730727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:40.552253 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.552215    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:40.552764 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.552565    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" podUID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" containerName="maas-api" containerID="cri-o://7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751" gracePeriod=30
Jul 23 09:49:40.806073 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.806015    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:40.925044 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.925011    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") pod \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") "
Jul 23 09:49:40.925230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.925052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjjnz\" (UniqueName: \"kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz\") pod \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") "
Jul 23 09:49:40.925230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.925081    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca\") pod \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\" (UID: \"47afdb26-6303-42e4-a8f4-55e7c97d1f5a\") "
Jul 23 09:49:40.925488 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.925463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "47afdb26-6303-42e4-a8f4-55e7c97d1f5a" (UID: "47afdb26-6303-42e4-a8f4-55e7c97d1f5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:49:40.927164 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.927142    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "47afdb26-6303-42e4-a8f4-55e7c97d1f5a" (UID: "47afdb26-6303-42e4-a8f4-55e7c97d1f5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:49:40.927244 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:40.927216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz" (OuterVolumeSpecName: "kube-api-access-gjjnz") pod "47afdb26-6303-42e4-a8f4-55e7c97d1f5a" (UID: "47afdb26-6303-42e4-a8f4-55e7c97d1f5a"). InnerVolumeSpecName "kube-api-access-gjjnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:49:41.026548 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.026516    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:41.026548 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.026546    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjnz\" (UniqueName: \"kubernetes.io/projected/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-kube-api-access-gjjnz\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:41.026739 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.026556    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47afdb26-6303-42e4-a8f4-55e7c97d1f5a-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:41.360082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.360040    2560 generic.go:356] "Generic (PLEG): container finished" podID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" containerID="7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751" exitCode=0
Jul 23 09:49:41.360082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.360080    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"
Jul 23 09:49:41.360296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.360130    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" event={"ID":"47afdb26-6303-42e4-a8f4-55e7c97d1f5a","Type":"ContainerDied","Data":"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"}
Jul 23 09:49:41.360296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.360181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj" event={"ID":"47afdb26-6303-42e4-a8f4-55e7c97d1f5a","Type":"ContainerDied","Data":"531116ba783185fdb7313cef60234a7fdcbb9f252291e6851e0346535a82ca54"}
Jul 23 09:49:41.360296 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.360207    2560 scope.go:117] "RemoveContainer" containerID="7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"
Jul 23 09:49:41.372258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.372235    2560 scope.go:117] "RemoveContainer" containerID="7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"
Jul 23 09:49:41.372719 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:49:41.372697    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751\": container with ID starting with 7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751 not found: ID does not exist" containerID="7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"
Jul 23 09:49:41.372807 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.372729    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751"} err="failed to get container status \"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751\": rpc error: code = NotFound desc = could not find container \"7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751\": container with ID starting with 7d8e1c47d1f727440b46417ee3daded22b6a32b6f7ad1f73d6c67dc4947d1751 not found: ID does not exist"
Jul 23 09:49:41.383921 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.383885    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:41.387238 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.387214    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18ba59c9-95c5954c8-7nnlj"]
Jul 23 09:49:41.912974 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:41.912943    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" path="/var/lib/kubelet/pods/47afdb26-6303-42e4-a8f4-55e7c97d1f5a/volumes"
Jul 23 09:49:52.131825 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:52.131741    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:52.132311 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:52.131981    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy" containerID="cri-o://335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18" gracePeriod=30
Jul 23 09:49:57.379555 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.379526    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:57.441764 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.441726    2560 generic.go:356] "Generic (PLEG): container finished" podID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerID="335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18" exitCode=0
Jul 23 09:49:57.441951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.441778    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"
Jul 23 09:49:57.441951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.441799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerDied","Data":"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"}
Jul 23 09:49:57.441951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.441875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm" event={"ID":"18c94fbe-ee64-4978-9029-b8ab4432f55d","Type":"ContainerDied","Data":"6b9e6035cafad675b9c7d39befa2ff90fcadcdcba722e63e5f843ccbd0014c9e"}
Jul 23 09:49:57.441951 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.441901    2560 scope.go:117] "RemoveContainer" containerID="335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"
Jul 23 09:49:57.450457 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.450438    2560 scope.go:117] "RemoveContainer" containerID="5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"
Jul 23 09:49:57.458483 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.458468    2560 scope.go:117] "RemoveContainer" containerID="335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"
Jul 23 09:49:57.458745 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:49:57.458716    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18\": container with ID starting with 335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18 not found: ID does not exist" containerID="335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"
Jul 23 09:49:57.458794 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.458753    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18"} err="failed to get container status \"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18\": rpc error: code = NotFound desc = could not find container \"335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18\": container with ID starting with 335316e895e031871445bce92cdadc6023049baa9ba9fa2f6f0e738e955dfe18 not found: ID does not exist"
Jul 23 09:49:57.458794 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.458769    2560 scope.go:117] "RemoveContainer" containerID="5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"
Jul 23 09:49:57.459021 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:49:57.459002    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1\": container with ID starting with 5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1 not found: ID does not exist" containerID="5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"
Jul 23 09:49:57.459081 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.459026    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1"} err="failed to get container status \"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1\": rpc error: code = NotFound desc = could not find container \"5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1\": container with ID starting with 5e9d81e71707d27b17d38ed89d377690cd284944aa0dc3f7249ad48b732390e1 not found: ID does not exist"
Jul 23 09:49:57.479331 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479312    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479407 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479407 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479371    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479407 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6s6s\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479450    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479472    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479531    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479560    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy\") pod \"18c94fbe-ee64-4978-9029-b8ab4432f55d\" (UID: \"18c94fbe-ee64-4978-9029-b8ab4432f55d\") "
Jul 23 09:49:57.479743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:49:57.479743 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:49:57.479908 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:49:57.479908 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479870    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.479908 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.479887    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.480092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.480055    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:49:57.480322 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.480300    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data" (OuterVolumeSpecName: "istio-data") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:49:57.481526 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.481507    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token" (OuterVolumeSpecName: "istio-token") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:49:57.481656 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.481636    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:49:57.482009 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.481988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:49:57.482076 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.481997    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s" (OuterVolumeSpecName: "kube-api-access-c6s6s") pod "18c94fbe-ee64-4978-9029-b8ab4432f55d" (UID: "18c94fbe-ee64-4978-9029-b8ab4432f55d"). InnerVolumeSpecName "kube-api-access-c6s6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:49:57.580873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580849    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c94fbe-ee64-4978-9029-b8ab4432f55d-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.580873 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580870    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.581038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580882    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.581038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580891    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.581038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580899    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.581038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580907    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c94fbe-ee64-4978-9029-b8ab4432f55d-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.581038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.580934    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6s6s\" (UniqueName: \"kubernetes.io/projected/18c94fbe-ee64-4978-9029-b8ab4432f55d-kube-api-access-c6s6s\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:49:57.769711 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.769669    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:57.773067 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.773041    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18ba59c9-openshift-default-6fbcd8d44c-b2qlm"]
Jul 23 09:49:57.913122 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:49:57.913093    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" path="/var/lib/kubelet/pods/18c94fbe-ee64-4978-9029-b8ab4432f55d/volumes"
Jul 23 09:50:12.266029 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.265995    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:50:12.266479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266463    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:12.266479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266481    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:12.266574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266504    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" containerName="maas-api"
Jul 23 09:50:12.266574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266510    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" containerName="maas-api"
Jul 23 09:50:12.266574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266573    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:12.266679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266582    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:12.266679 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.266591    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47afdb26-6303-42e4-a8f4-55e7c97d1f5a" containerName="maas-api"
Jul 23 09:50:12.269880 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.269860    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.272370 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.272348    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:50:12.272476 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.272401    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18c2a482-openshift-default-dockercfg-hh8r2\""
Jul 23 09:50:12.282147 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.282122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:50:12.309793 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.309757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9bv\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.309954 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.309844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.309954 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.309936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310065 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.309990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.310065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310114 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.310097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310193 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.310127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310193 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.310144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.310193 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.310173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410586 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410586 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9bv\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.410792 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410769    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.410863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411299 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.411269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.411358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411416 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.411393    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411508 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.411436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.411591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.411574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.413151 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.413131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.413238 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.413219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.420038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.420019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9bv\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.420157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.420138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token\") pod \"e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") " pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.579983 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.579888    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:12.719935 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.719120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:50:12.722262 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:50:12.722226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ee9281_bc37_49fb_9267_9923f2dfa017.slice/crio-658fd1b8cba24fcc178e3aca19054f8c9cac44d5c51af3bc43ffbcf81701e08f WatchSource:0}: Error finding container 658fd1b8cba24fcc178e3aca19054f8c9cac44d5c51af3bc43ffbcf81701e08f: Status 404 returned error can't find the container with id 658fd1b8cba24fcc178e3aca19054f8c9cac44d5c51af3bc43ffbcf81701e08f
Jul 23 09:50:12.724277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.724244    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:12.724369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.724310    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:12.724369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:12.724361    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:13.522016 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:13.521977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerStarted","Data":"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"}
Jul 23 09:50:13.522413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:13.522034    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerStarted","Data":"658fd1b8cba24fcc178e3aca19054f8c9cac44d5c51af3bc43ffbcf81701e08f"}
Jul 23 09:50:13.545760 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:13.545699    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podStartSLOduration=1.5456830400000001 podStartE2EDuration="1.54568304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:50:13.544597233 +0000 UTC m=+1998.222815017" watchObservedRunningTime="2026-07-23 09:50:13.54568304 +0000 UTC m=+1998.223900822"
Jul 23 09:50:13.580771 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:13.580746    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:13.585987 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:13.585967    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:14.527168 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:14.527136    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:14.528091 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:14.528067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:15.701990 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.701958    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:50:15.702479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.702464    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:15.702479 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.702480    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c94fbe-ee64-4978-9029-b8ab4432f55d" containerName="istio-proxy"
Jul 23 09:50:15.705667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.705651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.708142 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.708118    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-18c2a482-serving-cert\""
Jul 23 09:50:15.708292 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.708214    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:50:15.721790 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.721758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:50:15.743276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.743237    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.743276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.743279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqn5w\" (UniqueName: \"kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.743458 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.743340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.844343 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.844304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.844516 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.844354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqn5w\" (UniqueName: \"kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.844516 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.844384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.844516 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:50:15.844453    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-serving-cert: secret "maas-api-e2e-mt-18c2a482-serving-cert" not found
Jul 23 09:50:15.844625 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:50:15.844518    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls podName:ae0cab03-29d9-484a-87cf-62dcd31165cf nodeName:}" failed. No retries permitted until 2026-07-23 09:50:16.344500946 +0000 UTC m=+2001.022718720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls") pod "maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" (UID: "ae0cab03-29d9-484a-87cf-62dcd31165cf") : secret "maas-api-e2e-mt-18c2a482-serving-cert" not found
Jul 23 09:50:15.845066 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.845048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:15.852714 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:15.852694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqn5w\" (UniqueName: \"kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:16.350254 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:16.350222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:16.352585 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:16.352563    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") pod \"maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:16.624457 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:16.624363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:16.771082 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:16.771055    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:50:16.772718 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:50:16.772677    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0cab03_29d9_484a_87cf_62dcd31165cf.slice/crio-4c88c2b4b397c2a4941626b27dfae2f577a941d206644c0a50cc23a6efc90d1a WatchSource:0}: Error finding container 4c88c2b4b397c2a4941626b27dfae2f577a941d206644c0a50cc23a6efc90d1a: Status 404 returned error can't find the container with id 4c88c2b4b397c2a4941626b27dfae2f577a941d206644c0a50cc23a6efc90d1a
Jul 23 09:50:16.774206 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:16.774183    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:50:17.549831 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:17.549728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" event={"ID":"ae0cab03-29d9-484a-87cf-62dcd31165cf","Type":"ContainerStarted","Data":"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"}
Jul 23 09:50:17.549831 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:17.549803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" event={"ID":"ae0cab03-29d9-484a-87cf-62dcd31165cf","Type":"ContainerStarted","Data":"4c88c2b4b397c2a4941626b27dfae2f577a941d206644c0a50cc23a6efc90d1a"}
Jul 23 09:50:17.550374 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:17.549945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:17.567630 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:17.567576    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" podStartSLOduration=2.049277369 podStartE2EDuration="2.567561737s" totalImagesPullingTime="518.284368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:50:17.566841632 +0000 UTC m=+2002.245059445" watchObservedRunningTime="2026-07-23 09:50:17.567561737 +0000 UTC m=+2002.245779520"
Jul 23 09:50:23.561818 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:23.561789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:50:30.012575 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.011775    2560 patch_prober.go:28] interesting pod/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:38970->10.133.0.59:15021: read: connection reset by peer" start-of-body=
Jul 23 09:50:30.012575 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.011844    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": read tcp 10.133.0.2:38970->10.133.0.59:15021: read: connection reset by peer"
Jul 23 09:50:30.618956 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.618898    2560 generic.go:356] "Generic (PLEG): container finished" podID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerID="28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258" exitCode=0
Jul 23 09:50:30.619123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.618961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerDied","Data":"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"}
Jul 23 09:50:30.619354 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.619340    2560 scope.go:117] "RemoveContainer" containerID="28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"
Jul 23 09:50:30.620370 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.620339    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:30.620427 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.620406    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:30.620463 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:30.620438    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:50:31.625238 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:31.625204    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerStarted","Data":"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"}
Jul 23 09:50:32.580475 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:32.580438    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:32.580661 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:32.580620    2560 patch_prober.go:28] interesting pod/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 23 09:50:32.580709 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:32.580669    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 23 09:50:33.580783 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:33.580750    2560 patch_prober.go:28] interesting pod/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 23 09:50:33.581177 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:33.580806    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 23 09:50:34.580654 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:34.580622    2560 patch_prober.go:28] interesting pod/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 23 09:50:34.580822 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:34.580684    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 23 09:50:35.581004 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:35.580962    2560 patch_prober.go:28] interesting pod/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 23 09:50:35.581405 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:35.581036    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 23 09:50:36.583757 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:36.583728    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:36.584185 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:36.584024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:50:36.584713 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:50:36.584693    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:51:14.431465 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.431418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:51:14.432095 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.431639    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" podUID="ae0cab03-29d9-484a-87cf-62dcd31165cf" containerName="maas-api" containerID="cri-o://0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f" gracePeriod=30
Jul 23 09:51:14.689540 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.689484    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:51:14.776111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.776087    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqn5w\" (UniqueName: \"kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w\") pod \"ae0cab03-29d9-484a-87cf-62dcd31165cf\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") "
Jul 23 09:51:14.776248 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.776155    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") pod \"ae0cab03-29d9-484a-87cf-62dcd31165cf\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") "
Jul 23 09:51:14.776248 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.776191    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca\") pod \"ae0cab03-29d9-484a-87cf-62dcd31165cf\" (UID: \"ae0cab03-29d9-484a-87cf-62dcd31165cf\") "
Jul 23 09:51:14.776611 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.776590    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae0cab03-29d9-484a-87cf-62dcd31165cf" (UID: "ae0cab03-29d9-484a-87cf-62dcd31165cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:51:14.778210 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.778183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae0cab03-29d9-484a-87cf-62dcd31165cf" (UID: "ae0cab03-29d9-484a-87cf-62dcd31165cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:51:14.778330 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.778266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w" (OuterVolumeSpecName: "kube-api-access-mqn5w") pod "ae0cab03-29d9-484a-87cf-62dcd31165cf" (UID: "ae0cab03-29d9-484a-87cf-62dcd31165cf"). InnerVolumeSpecName "kube-api-access-mqn5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:51:14.849514 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.849484    2560 generic.go:356] "Generic (PLEG): container finished" podID="ae0cab03-29d9-484a-87cf-62dcd31165cf" containerID="0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f" exitCode=0
Jul 23 09:51:14.849614 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.849525    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"
Jul 23 09:51:14.849614 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.849568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" event={"ID":"ae0cab03-29d9-484a-87cf-62dcd31165cf","Type":"ContainerDied","Data":"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"}
Jul 23 09:51:14.849693 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.849621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g" event={"ID":"ae0cab03-29d9-484a-87cf-62dcd31165cf","Type":"ContainerDied","Data":"4c88c2b4b397c2a4941626b27dfae2f577a941d206644c0a50cc23a6efc90d1a"}
Jul 23 09:51:14.849693 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.849643    2560 scope.go:117] "RemoveContainer" containerID="0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"
Jul 23 09:51:14.858812 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.858797    2560 scope.go:117] "RemoveContainer" containerID="0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"
Jul 23 09:51:14.859081 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:51:14.859063    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f\": container with ID starting with 0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f not found: ID does not exist" containerID="0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"
Jul 23 09:51:14.859127 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.859091    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f"} err="failed to get container status \"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f\": rpc error: code = NotFound desc = could not find container \"0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f\": container with ID starting with 0f5e0d031a85db46aed0ac86fffe99727c8df7c44c71e59f9a9840a0f581cf5f not found: ID does not exist"
Jul 23 09:51:14.870512 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.870488    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:51:14.872413 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.872390    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-18c2a482-7fb6bb4bff-mzx9g"]
Jul 23 09:51:14.877545 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.877529    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqn5w\" (UniqueName: \"kubernetes.io/projected/ae0cab03-29d9-484a-87cf-62dcd31165cf-kube-api-access-mqn5w\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:51:14.877607 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.877551    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae0cab03-29d9-484a-87cf-62dcd31165cf-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:51:14.877607 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:14.877565    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae0cab03-29d9-484a-87cf-62dcd31165cf-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:51:15.919233 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:15.919199    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae0cab03-29d9-484a-87cf-62dcd31165cf" path="/var/lib/kubelet/pods/ae0cab03-29d9-484a-87cf-62dcd31165cf/volumes"
Jul 23 09:51:56.021699 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:56.021664    2560 scope.go:117] "RemoveContainer" containerID="24fdb1ddb4e52436f39d497b7ff48ac7f4a6c75f8d2035d636ef0afd47d66238"
Jul 23 09:51:56.029219 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:51:56.029203    2560 scope.go:117] "RemoveContainer" containerID="26a6438533c18a0ba3b065f6086993a2a7c7978b67dc291d771b6a9e9cfe253c"
Jul 23 09:52:08.801226 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:08.801190    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:52:08.801646 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:08.801420    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy" containerID="cri-o://c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f" gracePeriod=30
Jul 23 09:52:14.036597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.036571    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:52:14.149674 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.149598    2560 generic.go:356] "Generic (PLEG): container finished" podID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerID="c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f" exitCode=0
Jul 23 09:52:14.149674 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.149640    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"
Jul 23 09:52:14.149850 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.149685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerDied","Data":"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"}
Jul 23 09:52:14.149850 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.149736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm" event={"ID":"38ee9281-bc37-49fb-9267-9923f2dfa017","Type":"ContainerDied","Data":"658fd1b8cba24fcc178e3aca19054f8c9cac44d5c51af3bc43ffbcf81701e08f"}
Jul 23 09:52:14.149850 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.149758    2560 scope.go:117] "RemoveContainer" containerID="c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"
Jul 23 09:52:14.157499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157483    2560 scope.go:117] "RemoveContainer" containerID="28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"
Jul 23 09:52:14.157610 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157586    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157713 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn9bv\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157713 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157659    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157713 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157869 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157869 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:52:14.157869 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157803    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.157869 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157847    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.158111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157928    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.158111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:52:14.158111 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.157954    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy\") pod \"38ee9281-bc37-49fb-9267-9923f2dfa017\" (UID: \"38ee9281-bc37-49fb-9267-9923f2dfa017\") "
Jul 23 09:52:14.158332 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.158272    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:52:14.158414 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.158361    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.158414 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.158396    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.158511 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.158453    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:52:14.159132 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.159017    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data" (OuterVolumeSpecName: "istio-data") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:52:14.160020 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.159990    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:52:14.160196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.160173    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:52:14.160196 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.160187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv" (OuterVolumeSpecName: "kube-api-access-sn9bv") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "kube-api-access-sn9bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:52:14.160312 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.160203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token" (OuterVolumeSpecName: "istio-token") pod "38ee9281-bc37-49fb-9267-9923f2dfa017" (UID: "38ee9281-bc37-49fb-9267-9923f2dfa017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:52:14.178250 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.178230    2560 scope.go:117] "RemoveContainer" containerID="c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"
Jul 23 09:52:14.178461 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:52:14.178440    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f\": container with ID starting with c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f not found: ID does not exist" containerID="c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"
Jul 23 09:52:14.178510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.178467    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f"} err="failed to get container status \"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f\": rpc error: code = NotFound desc = could not find container \"c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f\": container with ID starting with c7ff8440d7e7c567c7275179507220554a8cd3edb2c2b646c850997ef6d51e8f not found: ID does not exist"
Jul 23 09:52:14.178510 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.178484    2560 scope.go:117] "RemoveContainer" containerID="28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"
Jul 23 09:52:14.178699 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:52:14.178685    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258\": container with ID starting with 28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258 not found: ID does not exist" containerID="28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"
Jul 23 09:52:14.178758 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.178702    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258"} err="failed to get container status \"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258\": rpc error: code = NotFound desc = could not find container \"28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258\": container with ID starting with 28bedb5dc81ca613fe8374df56bd4cb8be33cfc242a1a258fb0e006dc6c1a258 not found: ID does not exist"
Jul 23 09:52:14.259364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259339    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259366    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259379    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259390    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259402    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee9281-bc37-49fb-9267-9923f2dfa017-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259413    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee9281-bc37-49fb-9267-9923f2dfa017-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.259481 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.259424    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9bv\" (UniqueName: \"kubernetes.io/projected/38ee9281-bc37-49fb-9267-9923f2dfa017-kube-api-access-sn9bv\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:52:14.476023 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.475998    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:52:14.479665 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:14.479645    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18c2a482-openshift-default-5c68994fdc-xgdkm"]
Jul 23 09:52:15.913528 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:52:15.913494    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" path="/var/lib/kubelet/pods/38ee9281-bc37-49fb-9267-9923f2dfa017/volumes"
Jul 23 09:53:03.665571 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.665538    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:53:03.665988 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.665950    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.665988 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.665963    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.665988 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.665980    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.665988 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.665985    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.666121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.666006    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae0cab03-29d9-484a-87cf-62dcd31165cf" containerName="maas-api"
Jul 23 09:53:03.666121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.666012    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae0cab03-29d9-484a-87cf-62dcd31165cf" containerName="maas-api"
Jul 23 09:53:03.666121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.666067    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae0cab03-29d9-484a-87cf-62dcd31165cf" containerName="maas-api"
Jul 23 09:53:03.666121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.666082    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.666121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.666089    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ee9281-bc37-49fb-9267-9923f2dfa017" containerName="istio-proxy"
Jul 23 09:53:03.668959 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.668942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.671502 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.671479    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:53:03.671614 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.671532    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-16017a-openshift-default-dockercfg-49gfb\""
Jul 23 09:53:03.679569 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.679519    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:53:03.777925 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.777893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.777969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778262 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778262 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xgf\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778262 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778262 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.778262 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.778246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xgf\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.878774 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879055 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.878935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.879134    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879212 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.879180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879319 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.879246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879412 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.879390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.879532 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.879514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.881429 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.881405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.881582 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.881568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.886562 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.886541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xgf\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.886843 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.886822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token\") pod \"e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") " pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:03.979698 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:03.979643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:04.112444 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.112383    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:53:04.114649 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:53:04.114621    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2896263e_cb6d_44af_82d1_6ae3281ef6fb.slice/crio-cbae0bf953800a35f402924d001d8857b75b3e5e3bc1fc9895d4673556145a7f WatchSource:0}: Error finding container cbae0bf953800a35f402924d001d8857b75b3e5e3bc1fc9895d4673556145a7f: Status 404 returned error can't find the container with id cbae0bf953800a35f402924d001d8857b75b3e5e3bc1fc9895d4673556145a7f
Jul 23 09:53:04.116717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.116683    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:04.116822 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.116744    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:04.116822 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.116773    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:04.385419 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.385333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerStarted","Data":"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"}
Jul 23 09:53:04.385419 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.385390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerStarted","Data":"cbae0bf953800a35f402924d001d8857b75b3e5e3bc1fc9895d4673556145a7f"}
Jul 23 09:53:04.406418 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.406376    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podStartSLOduration=1.406361932 podStartE2EDuration="1.406361932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:53:04.404580576 +0000 UTC m=+2169.082798357" watchObservedRunningTime="2026-07-23 09:53:04.406361932 +0000 UTC m=+2169.084579713"
Jul 23 09:53:04.980276 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:04.980245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:05.985186 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:05.985158    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:06.394274 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.394185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:06.395028 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.395006    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:06.432809 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.432777    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:53:06.437366 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.437340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.439827 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.439801    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-16017a-serving-cert\""
Jul 23 09:53:06.440097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.440072    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:53:06.448576 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.448552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:53:06.503392 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.503357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.503539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.503398    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6cx7\" (UniqueName: \"kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.503698 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.503673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.604717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.604670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.604717 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.604718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6cx7\" (UniqueName: \"kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.605003 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.604775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.605361 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.605342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.607029 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.607013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.612864 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.612843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6cx7\" (UniqueName: \"kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7\") pod \"maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.754489 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.754464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:06.897590 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:06.897562    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:53:06.900299 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:53:06.900270    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9752a3_05a0_4b9a_be17_4f8e89341fb3.slice/crio-664e1322ff3f1344cdc9073227e6c6751ad2c3aed4b41476999b188c97df10df WatchSource:0}: Error finding container 664e1322ff3f1344cdc9073227e6c6751ad2c3aed4b41476999b188c97df10df: Status 404 returned error can't find the container with id 664e1322ff3f1344cdc9073227e6c6751ad2c3aed4b41476999b188c97df10df
Jul 23 09:53:07.401834 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:07.401790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" event={"ID":"4a9752a3-05a0-4b9a-be17-4f8e89341fb3","Type":"ContainerStarted","Data":"664e1322ff3f1344cdc9073227e6c6751ad2c3aed4b41476999b188c97df10df"}
Jul 23 09:53:08.408870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:08.408835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" event={"ID":"4a9752a3-05a0-4b9a-be17-4f8e89341fb3","Type":"ContainerStarted","Data":"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"}
Jul 23 09:53:08.409281 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:08.408906    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:08.425865 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:08.425818    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" podStartSLOduration=1.941618353 podStartE2EDuration="2.425803517s" totalImagesPullingTime="484.185164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:53:08.423935531 +0000 UTC m=+2173.102153335" watchObservedRunningTime="2026-07-23 09:53:08.425803517 +0000 UTC m=+2173.104021299"
Jul 23 09:53:14.421175 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:14.421146    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:53:14.913042 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:14.913015    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:53:14.917021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:14.916994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:14.919541 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:14.919511    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-64f44e-serving-cert\""
Jul 23 09:53:14.927927 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:14.927885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:53:15.075557 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.075527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.075728 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.075568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.075728 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.075610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl485\" (UniqueName: \"kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.177079 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.177007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.177079 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.177044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.177244 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.177101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl485\" (UniqueName: \"kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.177708 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.177690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.179324 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.179306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.185021 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.185000    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl485\" (UniqueName: \"kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485\") pod \"maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.230436 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.230405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:15.406740 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.406708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:53:15.408561 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:53:15.408525    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c9c609_61e0_49a9_afe8_80780f00b027.slice/crio-f9a4f24529e404a61b53dbe8370b7d31af42ac3ca29f1f4dc16532f367301f82 WatchSource:0}: Error finding container f9a4f24529e404a61b53dbe8370b7d31af42ac3ca29f1f4dc16532f367301f82: Status 404 returned error can't find the container with id f9a4f24529e404a61b53dbe8370b7d31af42ac3ca29f1f4dc16532f367301f82
Jul 23 09:53:15.462600 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:15.462522    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" event={"ID":"b8c9c609-61e0-49a9-afe8-80780f00b027","Type":"ContainerStarted","Data":"f9a4f24529e404a61b53dbe8370b7d31af42ac3ca29f1f4dc16532f367301f82"}
Jul 23 09:53:16.468283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:16.468248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" event={"ID":"b8c9c609-61e0-49a9-afe8-80780f00b027","Type":"ContainerStarted","Data":"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"}
Jul 23 09:53:16.468762 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:16.468363    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:16.487992 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:16.487943    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" podStartSLOduration=2.120296281 podStartE2EDuration="2.487929513s" totalImagesPullingTime="367.633232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:53:16.486167762 +0000 UTC m=+2181.164385571" watchObservedRunningTime="2026-07-23 09:53:16.487929513 +0000 UTC m=+2181.166147289"
Jul 23 09:53:20.493389 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.493351    2560 generic.go:356] "Generic (PLEG): container finished" podID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerID="92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460" exitCode=0
Jul 23 09:53:20.493857 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.493422    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerDied","Data":"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"}
Jul 23 09:53:20.493857 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.493832    2560 scope.go:117] "RemoveContainer" containerID="92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"
Jul 23 09:53:20.494824 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.494792    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:20.494935 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.494861    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:20.494935 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:20.494892    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:53:21.394955 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:21.394921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:21.500319 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:21.500282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerStarted","Data":"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"}
Jul 23 09:53:21.980097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:21.980057    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:21.981275 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:21.981247    2560 patch_prober.go:28] interesting pod/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 23 09:53:21.981401 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:21.981297    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 23 09:53:22.480645 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:22.480614    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:53:22.980801 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:22.980760    2560 patch_prober.go:28] interesting pod/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 23 09:53:22.981210 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:22.980824    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 23 09:53:23.980852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:23.980819    2560 patch_prober.go:28] interesting pod/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 23 09:53:23.981285 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:23.980879    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 23 09:53:24.980519 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:24.980485    2560 patch_prober.go:28] interesting pod/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Jul 23 09:53:24.980709 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:24.980547    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Jul 23 09:53:25.984112 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:25.984075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:25.984499 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:25.984457    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:53:25.985064 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:53:25.985042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:54:45.558421 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.558385    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:54:45.558989 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.558593    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" podUID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" containerName="maas-api" containerID="cri-o://9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c" gracePeriod=30
Jul 23 09:54:45.810309 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.810247    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:54:45.884455 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.884428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls\") pod \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") "
Jul 23 09:54:45.884587 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.884487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6cx7\" (UniqueName: \"kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7\") pod \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") "
Jul 23 09:54:45.884692 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.884668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca\") pod \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\" (UID: \"4a9752a3-05a0-4b9a-be17-4f8e89341fb3\") "
Jul 23 09:54:45.885024 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.885000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a9752a3-05a0-4b9a-be17-4f8e89341fb3" (UID: "4a9752a3-05a0-4b9a-be17-4f8e89341fb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:54:45.886429 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.886400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7" (OuterVolumeSpecName: "kube-api-access-h6cx7") pod "4a9752a3-05a0-4b9a-be17-4f8e89341fb3" (UID: "4a9752a3-05a0-4b9a-be17-4f8e89341fb3"). InnerVolumeSpecName "kube-api-access-h6cx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:54:45.886429 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.886417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a9752a3-05a0-4b9a-be17-4f8e89341fb3" (UID: "4a9752a3-05a0-4b9a-be17-4f8e89341fb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:54:45.922806 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.922785    2560 generic.go:356] "Generic (PLEG): container finished" podID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" containerID="9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c" exitCode=0
Jul 23 09:54:45.922902 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.922818    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"
Jul 23 09:54:45.922902 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.922864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" event={"ID":"4a9752a3-05a0-4b9a-be17-4f8e89341fb3","Type":"ContainerDied","Data":"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"}
Jul 23 09:54:45.922995 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.922908    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw" event={"ID":"4a9752a3-05a0-4b9a-be17-4f8e89341fb3","Type":"ContainerDied","Data":"664e1322ff3f1344cdc9073227e6c6751ad2c3aed4b41476999b188c97df10df"}
Jul 23 09:54:45.922995 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.922951    2560 scope.go:117] "RemoveContainer" containerID="9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"
Jul 23 09:54:45.933929 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.933872    2560 scope.go:117] "RemoveContainer" containerID="9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"
Jul 23 09:54:45.934219 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:54:45.934197    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c\": container with ID starting with 9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c not found: ID does not exist" containerID="9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"
Jul 23 09:54:45.934310 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.934228    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c"} err="failed to get container status \"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c\": rpc error: code = NotFound desc = could not find container \"9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c\": container with ID starting with 9ec35bbcf42612e984430d08a3e887e948a2450971cdc17fe051bb1edb2c4b2c not found: ID does not exist"
Jul 23 09:54:45.944315 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.944289    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:54:45.948287 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.948263    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-16017a-5c9c5d78b8-bm7qw"]
Jul 23 09:54:45.985891 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.985867    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:54:45.985995 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.985940    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6cx7\" (UniqueName: \"kubernetes.io/projected/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-kube-api-access-h6cx7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:54:45.985995 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:45.985955    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9752a3-05a0-4b9a-be17-4f8e89341fb3-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:54:47.913275 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:47.913242    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" path="/var/lib/kubelet/pods/4a9752a3-05a0-4b9a-be17-4f8e89341fb3/volumes"
Jul 23 09:54:58.001382 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:58.001347    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:54:58.001889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:54:58.001536    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy" containerID="cri-o://9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1" gracePeriod=30
Jul 23 09:55:03.261736 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.261713    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:55:03.442940 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.442897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6xgf\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.442961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.442998    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443043    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443249 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443114    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443249 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443249 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443202    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443249 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443266    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket\") pod \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\" (UID: \"2896263e-cb6d-44af-82d1-6ae3281ef6fb\") "
Jul 23 09:55:03.443443 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443375    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:55:03.443539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:55:03.443722 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443680    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:55:03.443722 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443705    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.443852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443728    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.443852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.443792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:55:03.444106 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.444083    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:55:03.445327 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.445289    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:55:03.445423 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.445381    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:55:03.445594 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.445577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf" (OuterVolumeSpecName: "kube-api-access-x6xgf") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "kube-api-access-x6xgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:55:03.445719 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.445698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2896263e-cb6d-44af-82d1-6ae3281ef6fb" (UID: "2896263e-cb6d-44af-82d1-6ae3281ef6fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:55:03.544745 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544723    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544745 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544745    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544754    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544762    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544770    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xgf\" (UniqueName: \"kubernetes.io/projected/2896263e-cb6d-44af-82d1-6ae3281ef6fb-kube-api-access-x6xgf\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544780    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:03.544872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:03.544793    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2896263e-cb6d-44af-82d1-6ae3281ef6fb-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:04.010761 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.010729    2560 generic.go:356] "Generic (PLEG): container finished" podID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerID="9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1" exitCode=0
Jul 23 09:55:04.010899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.010771    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"
Jul 23 09:55:04.010899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.010804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerDied","Data":"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"}
Jul 23 09:55:04.010899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.010859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g" event={"ID":"2896263e-cb6d-44af-82d1-6ae3281ef6fb","Type":"ContainerDied","Data":"cbae0bf953800a35f402924d001d8857b75b3e5e3bc1fc9895d4673556145a7f"}
Jul 23 09:55:04.010899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.010884    2560 scope.go:117] "RemoveContainer" containerID="9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"
Jul 23 09:55:04.019640 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.019623    2560 scope.go:117] "RemoveContainer" containerID="92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"
Jul 23 09:55:04.027123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.027106    2560 scope.go:117] "RemoveContainer" containerID="9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"
Jul 23 09:55:04.027360 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:55:04.027342    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1\": container with ID starting with 9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1 not found: ID does not exist" containerID="9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"
Jul 23 09:55:04.027399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.027368    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1"} err="failed to get container status \"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1\": rpc error: code = NotFound desc = could not find container \"9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1\": container with ID starting with 9dd67366f2dc20299c8bc6dc078a5de5af61023b3a4b09a214229b94c365acc1 not found: ID does not exist"
Jul 23 09:55:04.027399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.027384    2560 scope.go:117] "RemoveContainer" containerID="92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"
Jul 23 09:55:04.027636 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:55:04.027617    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460\": container with ID starting with 92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460 not found: ID does not exist" containerID="92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"
Jul 23 09:55:04.027672 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.027643    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460"} err="failed to get container status \"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460\": rpc error: code = NotFound desc = could not find container \"92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460\": container with ID starting with 92528fe329cf2b896aaba73c1a2b633d675c6a934c9fb06a018db1b29e2d2460 not found: ID does not exist"
Jul 23 09:55:04.034026 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.034004    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:55:04.041433 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:04.041412    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-16017a-openshift-default-6946b4999c-cxg7g"]
Jul 23 09:55:05.912705 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:05.912672    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" path="/var/lib/kubelet/pods/2896263e-cb6d-44af-82d1-6ae3281ef6fb/volumes"
Jul 23 09:55:09.111899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.111858    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:55:09.112415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.112078    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" podUID="b8c9c609-61e0-49a9-afe8-80780f00b027" containerName="maas-api" containerID="cri-o://43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788" gracePeriod=30
Jul 23 09:55:09.364893 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.364836    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:55:09.494367 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.494343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls\") pod \"b8c9c609-61e0-49a9-afe8-80780f00b027\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") "
Jul 23 09:55:09.494505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.494376    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl485\" (UniqueName: \"kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485\") pod \"b8c9c609-61e0-49a9-afe8-80780f00b027\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") "
Jul 23 09:55:09.494505 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.494442    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca\") pod \"b8c9c609-61e0-49a9-afe8-80780f00b027\" (UID: \"b8c9c609-61e0-49a9-afe8-80780f00b027\") "
Jul 23 09:55:09.494880 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.494848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8c9c609-61e0-49a9-afe8-80780f00b027" (UID: "b8c9c609-61e0-49a9-afe8-80780f00b027"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:55:09.496358 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.496330    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8c9c609-61e0-49a9-afe8-80780f00b027" (UID: "b8c9c609-61e0-49a9-afe8-80780f00b027"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:55:09.496471 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.496451    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485" (OuterVolumeSpecName: "kube-api-access-rl485") pod "b8c9c609-61e0-49a9-afe8-80780f00b027" (UID: "b8c9c609-61e0-49a9-afe8-80780f00b027"). InnerVolumeSpecName "kube-api-access-rl485". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:55:09.595852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.595825    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c9c609-61e0-49a9-afe8-80780f00b027-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:09.595852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.595848    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8c9c609-61e0-49a9-afe8-80780f00b027-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:09.595852 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:09.595857    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl485\" (UniqueName: \"kubernetes.io/projected/b8c9c609-61e0-49a9-afe8-80780f00b027-kube-api-access-rl485\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:55:10.041978 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.041945    2560 generic.go:356] "Generic (PLEG): container finished" podID="b8c9c609-61e0-49a9-afe8-80780f00b027" containerID="43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788" exitCode=0
Jul 23 09:55:10.042123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.042011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" event={"ID":"b8c9c609-61e0-49a9-afe8-80780f00b027","Type":"ContainerDied","Data":"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"}
Jul 23 09:55:10.042123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.042047    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"
Jul 23 09:55:10.042123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.042056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc" event={"ID":"b8c9c609-61e0-49a9-afe8-80780f00b027","Type":"ContainerDied","Data":"f9a4f24529e404a61b53dbe8370b7d31af42ac3ca29f1f4dc16532f367301f82"}
Jul 23 09:55:10.042123 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.042084    2560 scope.go:117] "RemoveContainer" containerID="43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"
Jul 23 09:55:10.050718 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.050690    2560 scope.go:117] "RemoveContainer" containerID="43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"
Jul 23 09:55:10.050996 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:55:10.050970    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788\": container with ID starting with 43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788 not found: ID does not exist" containerID="43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"
Jul 23 09:55:10.051094 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.051001    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788"} err="failed to get container status \"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788\": rpc error: code = NotFound desc = could not find container \"43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788\": container with ID starting with 43fa0e1e894e93ad2d72bce2a4554fc6519a67c7f4df27fb22b5dffce8c0d788 not found: ID does not exist"
Jul 23 09:55:10.065887 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.065864    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:55:10.074995 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:10.074972    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-64f44e-88fb6dcb-d4jkc"]
Jul 23 09:55:11.913092 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:11.913054    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c9c609-61e0-49a9-afe8-80780f00b027" path="/var/lib/kubelet/pods/b8c9c609-61e0-49a9-afe8-80780f00b027/volumes"
Jul 23 09:55:21.763446 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763415    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:55:21.763926 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763898    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c9c609-61e0-49a9-afe8-80780f00b027" containerName="maas-api"
Jul 23 09:55:21.763970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763930    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c9c609-61e0-49a9-afe8-80780f00b027" containerName="maas-api"
Jul 23 09:55:21.763970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763949    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" containerName="maas-api"
Jul 23 09:55:21.763970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763955    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" containerName="maas-api"
Jul 23 09:55:21.763970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763965    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.763970 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763970    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763978    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.763986    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.764058    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9752a3-05a0-4b9a-be17-4f8e89341fb3" containerName="maas-api"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.764066    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c9c609-61e0-49a9-afe8-80780f00b027" containerName="maas-api"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.764072    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.764131 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.764078    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2896263e-cb6d-44af-82d1-6ae3281ef6fb" containerName="istio-proxy"
Jul 23 09:55:21.768724 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.768706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.772108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.772080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-62b18d-openshift-default-dockercfg-lcx2z\""
Jul 23 09:55:21.772333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.772085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:55:21.785596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785783 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785783 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6668\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785865 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.785865 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.786008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.786008 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.785941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.797525 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.797503    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:55:21.887089 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887241 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6668\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887241 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887241 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887353 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887406 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887462 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887522 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887522 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887502    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887632 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887726 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887782 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.887821 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.887782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.888180 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.888161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.889386 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.889358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.889750 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.889731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.897551 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.897523    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:21.897866 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:21.897849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6668\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668\") pod \"e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") " pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:22.079346 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:22.079275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:22.229280 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:22.229256    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:55:22.231189 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:55:22.231156    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6231800d_28c7_449b_b5a3_96fcc9552f0c.slice/crio-b872041f5999f0ba5ca37c01112922e90d9bb44f453ce9fa113366f8d8e383a9 WatchSource:0}: Error finding container b872041f5999f0ba5ca37c01112922e90d9bb44f453ce9fa113366f8d8e383a9: Status 404 returned error can't find the container with id b872041f5999f0ba5ca37c01112922e90d9bb44f453ce9fa113366f8d8e383a9
Jul 23 09:55:22.233281 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:22.233250    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:22.233377 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:22.233311    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:22.233377 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:22.233340    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:23.107889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:23.107856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerStarted","Data":"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"}
Jul 23 09:55:23.108361 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:23.107925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerStarted","Data":"b872041f5999f0ba5ca37c01112922e90d9bb44f453ce9fa113366f8d8e383a9"}
Jul 23 09:55:23.153258 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:23.153204    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podStartSLOduration=2.153189383 podStartE2EDuration="2.153189383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:55:23.151577522 +0000 UTC m=+2307.829795305" watchObservedRunningTime="2026-07-23 09:55:23.153189383 +0000 UTC m=+2307.831407165"
Jul 23 09:55:24.079700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:24.079671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:24.084348 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:24.084323    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:24.114377 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:24.114355    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:24.115280 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:24.115257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:25.444072 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.444041    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:55:25.447747 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.447729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.455269 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.455250    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-62b18d-serving-cert\""
Jul 23 09:55:25.455396 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.455276    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:55:25.466214 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.466188    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:55:25.519899 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.519870    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk84t\" (UniqueName: \"kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.520045 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.519908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.520045 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.520018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.620832 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.620804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk84t\" (UniqueName: \"kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.620982 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.620844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.620982 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.620865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.621436 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.621405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.623266 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.623243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.667263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.667238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk84t\" (UniqueName: \"kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t\") pod \"maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.757857 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.757824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:25.914424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.914401    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:55:25.916139 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:55:25.916114    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89a05db_f889_48e3_9d89_c20aaef621b0.slice/crio-ff750194c8e1d5d73add0eb4a13f6475f246843ce83fed415363d2af3515e7ba WatchSource:0}: Error finding container ff750194c8e1d5d73add0eb4a13f6475f246843ce83fed415363d2af3515e7ba: Status 404 returned error can't find the container with id ff750194c8e1d5d73add0eb4a13f6475f246843ce83fed415363d2af3515e7ba
Jul 23 09:55:25.917593 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:25.917542    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 09:55:26.125439 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:26.125409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" event={"ID":"c89a05db-f889-48e3-9d89-c20aaef621b0","Type":"ContainerStarted","Data":"ff750194c8e1d5d73add0eb4a13f6475f246843ce83fed415363d2af3515e7ba"}
Jul 23 09:55:27.132054 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:27.132020    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" event={"ID":"c89a05db-f889-48e3-9d89-c20aaef621b0","Type":"ContainerStarted","Data":"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"}
Jul 23 09:55:27.132452 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:27.132087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:33.143501 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.143472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:55:33.183931 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.183851    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" podStartSLOduration=7.613596227 podStartE2EDuration="8.183835656s" totalImagesPullingTime="570.239429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:55:27.163464245 +0000 UTC m=+2311.841682052" watchObservedRunningTime="2026-07-23 09:55:33.183835656 +0000 UTC m=+2317.862053439"
Jul 23 09:55:33.834697 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.834653    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:55:33.839329 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.839297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:33.860137 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.860113    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-10b365-serving-cert\""
Jul 23 09:55:33.876898 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.876859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:55:33.898738 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.898699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznl7\" (UniqueName: \"kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:33.898897 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.898861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:33.898988 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:33.898902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.000169 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.000131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.000364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.000174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.000364 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.000205    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznl7\" (UniqueName: \"kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.000364 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:55:34.000319    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-serving-cert: secret "maas-api-e2e-api-b-10b365-serving-cert" not found
Jul 23 09:55:34.000547 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:55:34.000410    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls podName:fee333c1-3d90-4d20-92b8-4a923c4fac3b nodeName:}" failed. No retries permitted until 2026-07-23 09:55:34.500388896 +0000 UTC m=+2319.178606685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls") pod "maas-api-e2e-api-b-10b365-66584f9f7c-6669j" (UID: "fee333c1-3d90-4d20-92b8-4a923c4fac3b") : secret "maas-api-e2e-api-b-10b365-serving-cert" not found
Jul 23 09:55:34.000811 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.000789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.025005 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.024977    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznl7\" (UniqueName: \"kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.505771 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.505739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.508163 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.508137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") pod \"maas-api-e2e-api-b-10b365-66584f9f7c-6669j\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.761870 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.761767    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:34.953354 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:34.947641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:55:35.176424 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:35.176378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" event={"ID":"fee333c1-3d90-4d20-92b8-4a923c4fac3b","Type":"ContainerStarted","Data":"e26fec77cf57f66d734489731db14b782b89956b515aa1f4d87c39c938135fda"}
Jul 23 09:55:36.183667 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:36.183626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" event={"ID":"fee333c1-3d90-4d20-92b8-4a923c4fac3b","Type":"ContainerStarted","Data":"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"}
Jul 23 09:55:36.184142 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:36.183744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:36.251293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:36.251205    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" podStartSLOduration=2.6721579269999998 podStartE2EDuration="3.251175877s" totalImagesPullingTime="579.01795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:55:36.242246658 +0000 UTC m=+2320.920464470" watchObservedRunningTime="2026-07-23 09:55:36.251175877 +0000 UTC m=+2320.929393661"
Jul 23 09:55:40.157105 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:40.157066    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 23 09:55:40.157525 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:40.157127    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 09:55:41.213512 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.213477    2560 generic.go:356] "Generic (PLEG): container finished" podID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerID="70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99" exitCode=0
Jul 23 09:55:41.213984 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.213550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerDied","Data":"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"}
Jul 23 09:55:41.214032 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.213994    2560 scope.go:117] "RemoveContainer" containerID="70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"
Jul 23 09:55:41.215038 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.215008    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:41.215097 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.215081    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:41.215139 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:41.215113    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:55:42.196403 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:42.196368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:55:42.221561 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:42.221495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerStarted","Data":"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"}
Jul 23 09:55:43.080513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:43.080472    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:43.080760 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:43.080736    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 23 09:55:43.080833 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:43.080788    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 23 09:55:44.080181 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:44.080147    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 23 09:55:44.080636 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:44.080207    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 23 09:55:45.080324 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:45.080276    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 23 09:55:45.080742 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:45.080338    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 23 09:55:46.080096 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:46.080056    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Jul 23 09:55:46.080297 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:46.080125    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Jul 23 09:55:47.083625 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:47.083594    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:47.084112 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:47.083845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:55:47.084737 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:55:47.084713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:56:01.445454 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.445418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:56:01.446174 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.445716    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" podUID="c89a05db-f889-48e3-9d89-c20aaef621b0" containerName="maas-api" containerID="cri-o://5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815" gracePeriod=30
Jul 23 09:56:01.700726 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.700657    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:56:01.787466 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.787429    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk84t\" (UniqueName: \"kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t\") pod \"c89a05db-f889-48e3-9d89-c20aaef621b0\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") "
Jul 23 09:56:01.787620 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.787534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls\") pod \"c89a05db-f889-48e3-9d89-c20aaef621b0\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") "
Jul 23 09:56:01.787620 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.787569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca\") pod \"c89a05db-f889-48e3-9d89-c20aaef621b0\" (UID: \"c89a05db-f889-48e3-9d89-c20aaef621b0\") "
Jul 23 09:56:01.788004 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.787980    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c89a05db-f889-48e3-9d89-c20aaef621b0" (UID: "c89a05db-f889-48e3-9d89-c20aaef621b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:56:01.789562 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.789537    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c89a05db-f889-48e3-9d89-c20aaef621b0" (UID: "c89a05db-f889-48e3-9d89-c20aaef621b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:56:01.789643 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.789600    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t" (OuterVolumeSpecName: "kube-api-access-jk84t") pod "c89a05db-f889-48e3-9d89-c20aaef621b0" (UID: "c89a05db-f889-48e3-9d89-c20aaef621b0"). InnerVolumeSpecName "kube-api-access-jk84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:56:01.889369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.889330    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk84t\" (UniqueName: \"kubernetes.io/projected/c89a05db-f889-48e3-9d89-c20aaef621b0-kube-api-access-jk84t\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:01.889369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.889363    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89a05db-f889-48e3-9d89-c20aaef621b0-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:01.889369 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:01.889372    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89a05db-f889-48e3-9d89-c20aaef621b0-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:02.358379 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.358344    2560 generic.go:356] "Generic (PLEG): container finished" podID="c89a05db-f889-48e3-9d89-c20aaef621b0" containerID="5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815" exitCode=0
Jul 23 09:56:02.358574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.358389    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"
Jul 23 09:56:02.358574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.358424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" event={"ID":"c89a05db-f889-48e3-9d89-c20aaef621b0","Type":"ContainerDied","Data":"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"}
Jul 23 09:56:02.358574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.358479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2" event={"ID":"c89a05db-f889-48e3-9d89-c20aaef621b0","Type":"ContainerDied","Data":"ff750194c8e1d5d73add0eb4a13f6475f246843ce83fed415363d2af3515e7ba"}
Jul 23 09:56:02.358574 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.358502    2560 scope.go:117] "RemoveContainer" containerID="5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"
Jul 23 09:56:02.367108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.367090    2560 scope.go:117] "RemoveContainer" containerID="5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"
Jul 23 09:56:02.367382 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:02.367353    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815\": container with ID starting with 5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815 not found: ID does not exist" containerID="5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"
Jul 23 09:56:02.367382 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.367378    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815"} err="failed to get container status \"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815\": rpc error: code = NotFound desc = could not find container \"5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815\": container with ID starting with 5d8d0899ac50ea8bd4a2529e19d950543d90a12fa4f08259b1a86905f6963815 not found: ID does not exist"
Jul 23 09:56:02.386541 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.386500    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:56:02.394007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:02.393968    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-62b18d-fb77dbbf6-658w2"]
Jul 23 09:56:03.913058 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:03.913029    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c89a05db-f889-48e3-9d89-c20aaef621b0" path="/var/lib/kubelet/pods/c89a05db-f889-48e3-9d89-c20aaef621b0/volumes"
Jul 23 09:56:13.038900 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:13.038863    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:56:13.039325 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:13.039158    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" containerID="cri-o://1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b" gracePeriod=30
Jul 23 09:56:17.086230 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:17.086191    2560 patch_prober.go:28] interesting pod/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 09:56:17.086597 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:17.086254    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 09:56:18.306711 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.306642    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:56:18.443298 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.443261    2560 generic.go:356] "Generic (PLEG): container finished" podID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerID="1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b" exitCode=0
Jul 23 09:56:18.443469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.443310    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"
Jul 23 09:56:18.443469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.443349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerDied","Data":"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"}
Jul 23 09:56:18.443469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.443406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv" event={"ID":"6231800d-28c7-449b-b5a3-96fcc9552f0c","Type":"ContainerDied","Data":"b872041f5999f0ba5ca37c01112922e90d9bb44f453ce9fa113366f8d8e383a9"}
Jul 23 09:56:18.443469 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.443431    2560 scope.go:117] "RemoveContainer" containerID="1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"
Jul 23 09:56:18.450172 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450149    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450292 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450292 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450199    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450399 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450457 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450400    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450457 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450567 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6668\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450567 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450567 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450531    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs\") pod \"6231800d-28c7-449b-b5a3-96fcc9552f0c\" (UID: \"6231800d-28c7-449b-b5a3-96fcc9552f0c\") "
Jul 23 09:56:18.450730 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450671    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:56:18.450730 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450676    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:56:18.451225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450907    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6231800d-28c7-449b-b5a3-96fcc9552f0c-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.451225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.450946    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.451225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.451176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data" (OuterVolumeSpecName: "istio-data") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:56:18.451225 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.451189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:56:18.451564 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.451352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:56:18.452881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.452757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 09:56:18.452881 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.452840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token" (OuterVolumeSpecName: "istio-token") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:56:18.456474 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.456451    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 09:56:18.456591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.456475    2560 scope.go:117] "RemoveContainer" containerID="70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"
Jul 23 09:56:18.456591 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.456475    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668" (OuterVolumeSpecName: "kube-api-access-n6668") pod "6231800d-28c7-449b-b5a3-96fcc9552f0c" (UID: "6231800d-28c7-449b-b5a3-96fcc9552f0c"). InnerVolumeSpecName "kube-api-access-n6668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:56:18.473520 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.473495    2560 scope.go:117] "RemoveContainer" containerID="1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"
Jul 23 09:56:18.473792 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:18.473775    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b\": container with ID starting with 1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b not found: ID does not exist" containerID="1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"
Jul 23 09:56:18.473844 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.473802    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b"} err="failed to get container status \"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b\": rpc error: code = NotFound desc = could not find container \"1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b\": container with ID starting with 1221d87b204e05864f63fe989a82d9d198b2827027633a55892078c3afc12e1b not found: ID does not exist"
Jul 23 09:56:18.473844 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.473824    2560 scope.go:117] "RemoveContainer" containerID="70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"
Jul 23 09:56:18.474079 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:18.474060    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99\": container with ID starting with 70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99 not found: ID does not exist" containerID="70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"
Jul 23 09:56:18.474128 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.474086    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99"} err="failed to get container status \"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99\": rpc error: code = NotFound desc = could not find container \"70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99\": container with ID starting with 70d62d9022fcc41be48f823080fc1cbddf4d2f1fbe1628de9282ee7cb3635a99 not found: ID does not exist"
Jul 23 09:56:18.551660 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551628    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551660 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551656    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551660 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551664    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551882 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551673    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551882 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551682    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6668\" (UniqueName: \"kubernetes.io/projected/6231800d-28c7-449b-b5a3-96fcc9552f0c-kube-api-access-n6668\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551882 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551691    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6231800d-28c7-449b-b5a3-96fcc9552f0c-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.551882 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.551699    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6231800d-28c7-449b-b5a3-96fcc9552f0c-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:18.766157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.766124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:56:18.770353 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:18.770326    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-62b18d-openshift-default-6ff9fb8489-44zdv"]
Jul 23 09:56:19.913381 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:19.913345    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" path="/var/lib/kubelet/pods/6231800d-28c7-449b-b5a3-96fcc9552f0c/volumes"
Jul 23 09:56:28.811410 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:28.811376    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:56:28.811895 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:28.811609    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" podUID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" containerName="maas-api" containerID="cri-o://50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05" gracePeriod=30
Jul 23 09:56:29.064798 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.064731    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:56:29.150050 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.150020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cznl7\" (UniqueName: \"kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7\") pod \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") "
Jul 23 09:56:29.150239 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.150069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca\") pod \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") "
Jul 23 09:56:29.150239 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.150097    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") pod \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\" (UID: \"fee333c1-3d90-4d20-92b8-4a923c4fac3b\") "
Jul 23 09:56:29.150419 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.150396    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fee333c1-3d90-4d20-92b8-4a923c4fac3b" (UID: "fee333c1-3d90-4d20-92b8-4a923c4fac3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 09:56:29.152300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.152269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7" (OuterVolumeSpecName: "kube-api-access-cznl7") pod "fee333c1-3d90-4d20-92b8-4a923c4fac3b" (UID: "fee333c1-3d90-4d20-92b8-4a923c4fac3b"). InnerVolumeSpecName "kube-api-access-cznl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 09:56:29.152300 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.152278    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fee333c1-3d90-4d20-92b8-4a923c4fac3b" (UID: "fee333c1-3d90-4d20-92b8-4a923c4fac3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 09:56:29.251696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.251666    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cznl7\" (UniqueName: \"kubernetes.io/projected/fee333c1-3d90-4d20-92b8-4a923c4fac3b-kube-api-access-cznl7\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:29.251696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.251692    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee333c1-3d90-4d20-92b8-4a923c4fac3b-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:29.251696 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.251702    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee333c1-3d90-4d20-92b8-4a923c4fac3b-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 09:56:29.501903 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.501863    2560 generic.go:356] "Generic (PLEG): container finished" podID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" containerID="50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05" exitCode=0
Jul 23 09:56:29.502155 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.501908    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"
Jul 23 09:56:29.502155 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.501954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" event={"ID":"fee333c1-3d90-4d20-92b8-4a923c4fac3b","Type":"ContainerDied","Data":"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"}
Jul 23 09:56:29.502155 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.502019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j" event={"ID":"fee333c1-3d90-4d20-92b8-4a923c4fac3b","Type":"ContainerDied","Data":"e26fec77cf57f66d734489731db14b782b89956b515aa1f4d87c39c938135fda"}
Jul 23 09:56:29.502155 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.502045    2560 scope.go:117] "RemoveContainer" containerID="50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"
Jul 23 09:56:29.511522 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.511503    2560 scope.go:117] "RemoveContainer" containerID="50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"
Jul 23 09:56:29.511800 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:29.511782    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05\": container with ID starting with 50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05 not found: ID does not exist" containerID="50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"
Jul 23 09:56:29.511872 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.511806    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05"} err="failed to get container status \"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05\": rpc error: code = NotFound desc = could not find container \"50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05\": container with ID starting with 50e4c3b137ed47ae63b8c04ffec53974a4c0bd380820d9d1ebd61c000834bf05 not found: ID does not exist"
Jul 23 09:56:29.529906 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.527456    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:56:29.533710 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.533677    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-10b365-66584f9f7c-6669j"]
Jul 23 09:56:29.912382 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:29.912348    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" path="/var/lib/kubelet/pods/fee333c1-3d90-4d20-92b8-4a923c4fac3b/volumes"
Jul 23 09:56:40.891568 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891532    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891930    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c89a05db-f889-48e3-9d89-c20aaef621b0" containerName="maas-api"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891945    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c89a05db-f889-48e3-9d89-c20aaef621b0" containerName="maas-api"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891974    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" containerName="maas-api"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891980    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" containerName="maas-api"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891988    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891993    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.891999    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.892052 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.892005    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.892303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.892061    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c89a05db-f889-48e3-9d89-c20aaef621b0" containerName="maas-api"
Jul 23 09:56:40.892303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.892068    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fee333c1-3d90-4d20-92b8-4a923c4fac3b" containerName="maas-api"
Jul 23 09:56:40.892303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.892074    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.892303 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.892080    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6231800d-28c7-449b-b5a3-96fcc9552f0c" containerName="istio-proxy"
Jul 23 09:56:40.895113 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.895098    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:40.897700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.897682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 09:56:40.897809 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.897684    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-efdc11-openshift-default-dockercfg-h5mwv\""
Jul 23 09:56:40.907277 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:40.907256    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 09:56:41.059124 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059213    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszbm\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059293 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059506 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059506 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.059506 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.059396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160453 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160636 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160636 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.160765 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161007 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.160843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszbm\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161263 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.161236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.161262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161415 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.161312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161547 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.161417    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.161709 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.161685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.162730 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.162700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.163121 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.163099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.179522 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.179490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszbm\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.180349 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.180328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token\") pod \"e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") " pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.205385 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.205363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:41.351157 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.351018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 09:56:41.353332 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:56:41.353305    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19323cd3_b3f1_4e49_8e8f_12a69d2def1a.slice/crio-3e9d280e83037f646096a9a4ef7fec6c640d67da179c5d0136148447408a7e71 WatchSource:0}: Error finding container 3e9d280e83037f646096a9a4ef7fec6c640d67da179c5d0136148447408a7e71: Status 404 returned error can't find the container with id 3e9d280e83037f646096a9a4ef7fec6c640d67da179c5d0136148447408a7e71
Jul 23 09:56:41.355491 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.355461    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:41.355561 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.355527    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:41.355599 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.355559    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:41.567342 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.567296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerStarted","Data":"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"}
Jul 23 09:56:41.567536 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.567352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerStarted","Data":"3e9d280e83037f646096a9a4ef7fec6c640d67da179c5d0136148447408a7e71"}
Jul 23 09:56:41.592464 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:41.592409    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podStartSLOduration=1.592392869 podStartE2EDuration="1.592392869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:56:41.589407999 +0000 UTC m=+2386.267625785" watchObservedRunningTime="2026-07-23 09:56:41.592392869 +0000 UTC m=+2386.270610652"
Jul 23 09:56:42.206087 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:42.206048    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:42.210982 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:42.210956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:42.572539 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:42.572451    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:42.573449 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:42.573427    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:43.694889 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.694858    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 09:56:43.698583 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.698566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.701294 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.701271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-efdc11-serving-cert\""
Jul 23 09:56:43.701414 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.701347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zh7zc\""
Jul 23 09:56:43.709771 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.709744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 09:56:43.782630 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.782598    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.782820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.782663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.782820 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.782691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6bn\" (UniqueName: \"kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.883535 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.883495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.883535 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.883535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6bn\" (UniqueName: \"kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.883748 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.883604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.883748 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:43.883735    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-serving-cert: secret "maas-api-e2e-shared-a-efdc11-serving-cert" not found
Jul 23 09:56:43.883815 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:43.883801    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls podName:78c9d82a-37ec-4deb-a268-4f9ac9636c6e nodeName:}" failed. No retries permitted until 2026-07-23 09:56:44.383782367 +0000 UTC m=+2389.062000150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls") pod "maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" (UID: "78c9d82a-37ec-4deb-a268-4f9ac9636c6e") : secret "maas-api-e2e-shared-a-efdc11-serving-cert" not found
Jul 23 09:56:43.884165 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.884146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:43.903566 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:43.903541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6bn\" (UniqueName: \"kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:44.386513 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:44.386476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:44.388944 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:44.388900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") pod \"maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:44.609795 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:44.609762    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:44.744353 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:44.744328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 09:56:44.746934 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:56:44.746883    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9d82a_37ec_4deb_a268_4f9ac9636c6e.slice/crio-857adf7f1c7761a1935663fc7048eca41330936f451af9908a2d64a6dd7ab6a2 WatchSource:0}: Error finding container 857adf7f1c7761a1935663fc7048eca41330936f451af9908a2d64a6dd7ab6a2: Status 404 returned error can't find the container with id 857adf7f1c7761a1935663fc7048eca41330936f451af9908a2d64a6dd7ab6a2
Jul 23 09:56:45.588949 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:45.588901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" event={"ID":"78c9d82a-37ec-4deb-a268-4f9ac9636c6e","Type":"ContainerStarted","Data":"7201d8af7305ca34d8e1429d0e0db9be7ef5efc46d099d6fc52834f3e3b6e9a2"}
Jul 23 09:56:45.589115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:45.588968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" event={"ID":"78c9d82a-37ec-4deb-a268-4f9ac9636c6e","Type":"ContainerStarted","Data":"857adf7f1c7761a1935663fc7048eca41330936f451af9908a2d64a6dd7ab6a2"}
Jul 23 09:56:45.589115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:45.589023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:45.606362 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:45.606273    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" podStartSLOduration=2.006953878 podStartE2EDuration="2.606257133s" totalImagesPullingTime="599.303255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:56:45.605750291 +0000 UTC m=+2390.283968105" watchObservedRunningTime="2026-07-23 09:56:45.606257133 +0000 UTC m=+2390.284474918"
Jul 23 09:56:51.599596 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:51.599563    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 09:56:52.004456 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.004420    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 09:56:52.008158 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.008143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.011700 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.011675    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-0dc985-serving-cert\""
Jul 23 09:56:52.022069 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.022045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 09:56:52.065108 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.065065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.065278 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.065230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krwb5\" (UniqueName: \"kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.065333 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.065272    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.165760 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.165719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krwb5\" (UniqueName: \"kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.165760 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.165762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.166081 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.165820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.166081 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:52.165973    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-serving-cert: secret "maas-api-e2e-shared-b-0dc985-serving-cert" not found
Jul 23 09:56:52.166081 ip-10-0-129-47 kubenswrapper[2560]: E0723 09:56:52.166052    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls podName:6104a6ef-55fc-4e00-b103-94fc61c07e34 nodeName:}" failed. No retries permitted until 2026-07-23 09:56:52.666031498 +0000 UTC m=+2397.344249284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls") pod "maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" (UID: "6104a6ef-55fc-4e00-b103-94fc61c07e34") : secret "maas-api-e2e-shared-b-0dc985-serving-cert" not found
Jul 23 09:56:52.166494 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.166471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.177991 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.177966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krwb5\" (UniqueName: \"kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.670466 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.670435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.672967 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.672941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") pod \"maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:52.918307 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:52.918272    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:53.095798 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:53.095675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 09:56:53.099881 ip-10-0-129-47 kubenswrapper[2560]: W0723 09:56:53.099854    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6104a6ef_55fc_4e00_b103_94fc61c07e34.slice/crio-4858a927a8de759bf6fb1f3f27a2891ffdcd6474b99eb449db5b0252be999ab0 WatchSource:0}: Error finding container 4858a927a8de759bf6fb1f3f27a2891ffdcd6474b99eb449db5b0252be999ab0: Status 404 returned error can't find the container with id 4858a927a8de759bf6fb1f3f27a2891ffdcd6474b99eb449db5b0252be999ab0
Jul 23 09:56:53.637283 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:53.637250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" event={"ID":"6104a6ef-55fc-4e00-b103-94fc61c07e34","Type":"ContainerStarted","Data":"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"}
Jul 23 09:56:53.637559 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:53.637302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" event={"ID":"6104a6ef-55fc-4e00-b103-94fc61c07e34","Type":"ContainerStarted","Data":"4858a927a8de759bf6fb1f3f27a2891ffdcd6474b99eb449db5b0252be999ab0"}
Jul 23 09:56:53.637559 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:53.637343    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:56:53.657720 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:53.657651    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" podStartSLOduration=2.297187657 podStartE2EDuration="2.657626185s" totalImagesPullingTime="360.438528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 09:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 09:56:53.653637244 +0000 UTC m=+2398.331855046" watchObservedRunningTime="2026-07-23 09:56:53.657626185 +0000 UTC m=+2398.335843969"
Jul 23 09:56:56.657452 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.657418    2560 generic.go:356] "Generic (PLEG): container finished" podID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerID="173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c" exitCode=0
Jul 23 09:56:56.657878 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.657500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerDied","Data":"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"}
Jul 23 09:56:56.657942 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.657886    2560 scope.go:117] "RemoveContainer" containerID="173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"
Jul 23 09:56:56.658972 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.658906    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:56.659099 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.659017    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:56.659099 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:56.659063    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 09:56:57.572643 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:57.572603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:57.665565 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:57.665526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerStarted","Data":"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"}
Jul 23 09:56:58.205571 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:58.205538    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:56:58.206529 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:58.206502    2560 patch_prober.go:28] interesting pod/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 23 09:56:58.206630 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:58.206549    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 23 09:56:59.206115 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:59.206082    2560 patch_prober.go:28] interesting pod/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 23 09:56:59.206545 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:59.206512    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 23 09:56:59.656059 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:56:59.655968    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 09:57:00.205791 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:00.205747    2560 patch_prober.go:28] interesting pod/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 23 09:57:00.205996 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:00.205820    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 23 09:57:01.206040 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:01.206003    2560 patch_prober.go:28] interesting pod/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Jul 23 09:57:01.206421 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:01.206058    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Jul 23 09:57:02.210171 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:02.210141    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:57:02.210527 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:02.210439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 09:57:02.211146 ip-10-0-129-47 kubenswrapper[2560]: I0723 09:57:02.211129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 10:03:06.398682 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.398635    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 10:03:06.399177 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.398870    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerName="maas-api" containerID="cri-o://7201d8af7305ca34d8e1429d0e0db9be7ef5efc46d099d6fc52834f3e3b6e9a2" gracePeriod=30
Jul 23 10:03:06.536322 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.536272    2560 generic.go:356] "Generic (PLEG): container finished" podID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerID="7201d8af7305ca34d8e1429d0e0db9be7ef5efc46d099d6fc52834f3e3b6e9a2" exitCode=0
Jul 23 10:03:06.536558 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.536362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" event={"ID":"78c9d82a-37ec-4deb-a268-4f9ac9636c6e","Type":"ContainerDied","Data":"7201d8af7305ca34d8e1429d0e0db9be7ef5efc46d099d6fc52834f3e3b6e9a2"}
Jul 23 10:03:06.669051 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.669018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 10:03:06.781906 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.781868    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r6bn\" (UniqueName: \"kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn\") pod \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") "
Jul 23 10:03:06.781906 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.781908    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca\") pod \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") "
Jul 23 10:03:06.782183 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.781996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") pod \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\" (UID: \"78c9d82a-37ec-4deb-a268-4f9ac9636c6e\") "
Jul 23 10:03:06.782358 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.782332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78c9d82a-37ec-4deb-a268-4f9ac9636c6e" (UID: "78c9d82a-37ec-4deb-a268-4f9ac9636c6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 10:03:06.784265 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.784234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn" (OuterVolumeSpecName: "kube-api-access-4r6bn") pod "78c9d82a-37ec-4deb-a268-4f9ac9636c6e" (UID: "78c9d82a-37ec-4deb-a268-4f9ac9636c6e"). InnerVolumeSpecName "kube-api-access-4r6bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 10:03:06.784433 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.784298    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78c9d82a-37ec-4deb-a268-4f9ac9636c6e" (UID: "78c9d82a-37ec-4deb-a268-4f9ac9636c6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 10:03:06.882893 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.882839    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:06.882893 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.882881    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r6bn\" (UniqueName: \"kubernetes.io/projected/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-kube-api-access-4r6bn\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:06.882893 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:06.882892    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9d82a-37ec-4deb-a268-4f9ac9636c6e-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:07.543595 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.543565    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"
Jul 23 10:03:07.543595 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.543564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" event={"ID":"78c9d82a-37ec-4deb-a268-4f9ac9636c6e","Type":"ContainerDied","Data":"857adf7f1c7761a1935663fc7048eca41330936f451af9908a2d64a6dd7ab6a2"}
Jul 23 10:03:07.544125 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.543642    2560 scope.go:117] "RemoveContainer" containerID="7201d8af7305ca34d8e1429d0e0db9be7ef5efc46d099d6fc52834f3e3b6e9a2"
Jul 23 10:03:07.567809 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.567769    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 10:03:07.573714 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.573680    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm"]
Jul 23 10:03:07.913273 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:07.913239    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" path="/var/lib/kubelet/pods/78c9d82a-37ec-4deb-a268-4f9ac9636c6e/volumes"
Jul 23 10:03:09.595082 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:09.595018    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-efdc11-8dbc4b5f8-prpkm" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.68:8443/health\": context deadline exceeded"
Jul 23 10:03:17.793608 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:17.793516    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 10:03:17.794066 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:17.793772    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy" containerID="cri-o://47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555" gracePeriod=30
Jul 23 10:03:23.056236 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.056158    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 10:03:23.125524 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125484    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125701 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125551    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125701 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125606    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gszbm\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125701 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125701 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125685    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125892 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125722    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125892 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125756    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125892 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125818    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.125892 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.125857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo\") pod \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\" (UID: \"19323cd3-b3f1-4e49-8e8f-12a69d2def1a\") "
Jul 23 10:03:23.126166 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126106    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 10:03:23.126263 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126158    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 10:03:23.126263 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126229    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 10:03:23.126263 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126235    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.126680 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 10:03:23.126680 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.126657    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 10:03:23.128031 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.127995    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 10:03:23.128140 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.128098    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm" (OuterVolumeSpecName: "kube-api-access-gszbm") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "kube-api-access-gszbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 10:03:23.128194 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.128145    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 10:03:23.128234 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.128210    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19323cd3-b3f1-4e49-8e8f-12a69d2def1a" (UID: "19323cd3-b3f1-4e49-8e8f-12a69d2def1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 10:03:23.227685 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227637    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszbm\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-kube-api-access-gszbm\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.227685 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227674    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istiod-ca-cert\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.227685 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227685    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-credential-socket\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.227685 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227695    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-workload-certs\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.228005 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227704    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-token\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.228005 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227713    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-podinfo\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.228005 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227720    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-data\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.228005 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.227729    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19323cd3-b3f1-4e49-8e8f-12a69d2def1a-istio-envoy\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:23.629111 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.629071    2560 generic.go:356] "Generic (PLEG): container finished" podID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerID="47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555" exitCode=0
Jul 23 10:03:23.629302 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.629130    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"
Jul 23 10:03:23.629302 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.629161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerDied","Data":"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"}
Jul 23 10:03:23.629302 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.629215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj" event={"ID":"19323cd3-b3f1-4e49-8e8f-12a69d2def1a","Type":"ContainerDied","Data":"3e9d280e83037f646096a9a4ef7fec6c640d67da179c5d0136148447408a7e71"}
Jul 23 10:03:23.629302 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.629239    2560 scope.go:117] "RemoveContainer" containerID="47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"
Jul 23 10:03:23.638743 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.638725    2560 scope.go:117] "RemoveContainer" containerID="173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"
Jul 23 10:03:23.647640 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.647620    2560 scope.go:117] "RemoveContainer" containerID="47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"
Jul 23 10:03:23.647976 ip-10-0-129-47 kubenswrapper[2560]: E0723 10:03:23.647955    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555\": container with ID starting with 47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555 not found: ID does not exist" containerID="47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"
Jul 23 10:03:23.648037 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.647987    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555"} err="failed to get container status \"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555\": rpc error: code = NotFound desc = could not find container \"47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555\": container with ID starting with 47699ee3402222683f72a7ed6a2723a76398e7c791648483735997f58b9c7555 not found: ID does not exist"
Jul 23 10:03:23.648037 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.648009    2560 scope.go:117] "RemoveContainer" containerID="173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"
Jul 23 10:03:23.648308 ip-10-0-129-47 kubenswrapper[2560]: E0723 10:03:23.648289    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c\": container with ID starting with 173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c not found: ID does not exist" containerID="173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"
Jul 23 10:03:23.648354 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.648313    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c"} err="failed to get container status \"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c\": rpc error: code = NotFound desc = could not find container \"173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c\": container with ID starting with 173e5c6719495f8ef3b43804018e39902edb604ce94f85f91451800a20e5a84c not found: ID does not exist"
Jul 23 10:03:23.653784 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.653753    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 10:03:23.659476 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.659444    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-efdc11-openshift-default-7767c88bf5-cl2qj"]
Jul 23 10:03:23.916679 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:23.916413    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" path="/var/lib/kubelet/pods/19323cd3-b3f1-4e49-8e8f-12a69d2def1a/volumes"
Jul 23 10:03:28.735263 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:28.735227    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 10:03:28.736014 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:28.735504    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" podUID="6104a6ef-55fc-4e00-b103-94fc61c07e34" containerName="maas-api" containerID="cri-o://d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b" gracePeriod=30
Jul 23 10:03:28.992753 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:28.992682    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 10:03:29.085830 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.085775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") pod \"6104a6ef-55fc-4e00-b103-94fc61c07e34\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") "
Jul 23 10:03:29.086049 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.085961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krwb5\" (UniqueName: \"kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5\") pod \"6104a6ef-55fc-4e00-b103-94fc61c07e34\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") "
Jul 23 10:03:29.086049 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.085990    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca\") pod \"6104a6ef-55fc-4e00-b103-94fc61c07e34\" (UID: \"6104a6ef-55fc-4e00-b103-94fc61c07e34\") "
Jul 23 10:03:29.086403 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.086373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6104a6ef-55fc-4e00-b103-94fc61c07e34" (UID: "6104a6ef-55fc-4e00-b103-94fc61c07e34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 10:03:29.087994 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.087966    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6104a6ef-55fc-4e00-b103-94fc61c07e34" (UID: "6104a6ef-55fc-4e00-b103-94fc61c07e34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 10:03:29.088095 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.088050    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5" (OuterVolumeSpecName: "kube-api-access-krwb5") pod "6104a6ef-55fc-4e00-b103-94fc61c07e34" (UID: "6104a6ef-55fc-4e00-b103-94fc61c07e34"). InnerVolumeSpecName "kube-api-access-krwb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 10:03:29.187100 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.187053    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krwb5\" (UniqueName: \"kubernetes.io/projected/6104a6ef-55fc-4e00-b103-94fc61c07e34-kube-api-access-krwb5\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:29.187100 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.187089    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6104a6ef-55fc-4e00-b103-94fc61c07e34-openshift-service-ca\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:29.187100 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.187100    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6104a6ef-55fc-4e00-b103-94fc61c07e34-maas-api-tls\") on node \"ip-10-0-129-47.ec2.internal\" DevicePath \"\""
Jul 23 10:03:29.665189 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.665146    2560 generic.go:356] "Generic (PLEG): container finished" podID="6104a6ef-55fc-4e00-b103-94fc61c07e34" containerID="d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b" exitCode=0
Jul 23 10:03:29.665394 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.665199    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"
Jul 23 10:03:29.665394 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.665201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" event={"ID":"6104a6ef-55fc-4e00-b103-94fc61c07e34","Type":"ContainerDied","Data":"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"}
Jul 23 10:03:29.665394 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.665270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw" event={"ID":"6104a6ef-55fc-4e00-b103-94fc61c07e34","Type":"ContainerDied","Data":"4858a927a8de759bf6fb1f3f27a2891ffdcd6474b99eb449db5b0252be999ab0"}
Jul 23 10:03:29.665394 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.665303    2560 scope.go:117] "RemoveContainer" containerID="d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"
Jul 23 10:03:29.675626 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.675606    2560 scope.go:117] "RemoveContainer" containerID="d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"
Jul 23 10:03:29.676002 ip-10-0-129-47 kubenswrapper[2560]: E0723 10:03:29.675982    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b\": container with ID starting with d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b not found: ID does not exist" containerID="d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"
Jul 23 10:03:29.676069 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.676012    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b"} err="failed to get container status \"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b\": rpc error: code = NotFound desc = could not find container \"d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b\": container with ID starting with d0a2022aa15e908ce5465e8da784fc95bbcdc4c218acea79e2e23c02a0046a8b not found: ID does not exist"
Jul 23 10:03:29.687243 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.687188    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 10:03:29.691725 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.691690    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-0dc985-57bbcbbc98-9b9fw"]
Jul 23 10:03:29.913315 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:29.913281    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6104a6ef-55fc-4e00-b103-94fc61c07e34" path="/var/lib/kubelet/pods/6104a6ef-55fc-4e00-b103-94fc61c07e34/volumes"
Jul 23 10:03:46.463571 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:46.463533    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c9f8fb88-8m69g_27950d6c-29fa-47e0-b465-8507f8a016c6/manager/0.log"
Jul 23 10:03:46.579140 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:46.579110    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8vjr4_bd2e00f1-3288-43a1-b3ee-c11dd12442db/server/0.log"
Jul 23 10:03:46.823960 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:46.823845    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-2765v_7051f641-18e2-44e6-80e3-b581c48b5066/copy-manifests/0.log"
Jul 23 10:03:46.842926 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:46.842880    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-2765v_7051f641-18e2-44e6-80e3-b581c48b5066/manager/0.log"
Jul 23 10:03:47.590252 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:47.590217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746650-5plfh_b75fa870-96d6-4035-b54e-91cdeead9da0/cleanup/0.log"
Jul 23 10:03:47.705192 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:47.705161    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746665-dgr7r_865a3daf-2d2a-4559-bdb0-e57dbf559250/cleanup/0.log"
Jul 23 10:03:47.938817 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:47.938789    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fk9b8_ee1e2ab8-0198-446c-89b9-249020a9f62f/postgres/0.log"
Jul 23 10:03:49.572336 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:49.572305    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7cgdk_1534f616-ae0c-4e0e-86a3-9dbed3d7e356/manager/0.log"
Jul 23 10:03:49.690126 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:49.690094    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-p5hbq_91e4a155-59b8-4e44-84c1-007a13e33cd5/kuadrant-console-plugin/0.log"
Jul 23 10:03:49.827216 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:49.827124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lb4gb_44ce8fa2-fd76-409b-8cca-5ffe1850f3bd/registry-server/0.log"
Jul 23 10:03:49.954962 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:49.954927    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7mzw2_d83f55d6-df56-4db5-9a46-df7abbbe1c0a/manager/0.log"
Jul 23 10:03:50.198249 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:50.198216    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lhn2f_985c8d9d-ff72-4873-a70d-e29994d39ed4/manager/0.log"
Jul 23 10:03:51.179742 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:51.179711    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-wrrg6_1bef368c-87c7-495b-87d5-3a834ebb9e15/payload-pre-processing/0.log"
Jul 23 10:03:51.312902 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:51.312870    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-x2k4v_5584fdd3-e2a9-429e-a7b2-8367834a10fa/payload-processing/0.log"
Jul 23 10:03:51.939397 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:51.939370    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wtcvb_eb18cf27-205e-4c4f-aad1-b5459a521ee8/main/0.log"
Jul 23 10:03:52.055609 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:52.055569    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mf67f_4898197e-f975-4dbd-9525-e557d8081c83/main/0.log"
Jul 23 10:03:58.164294 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164259    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmqp/must-gather-jsrh6"]
Jul 23 10:03:58.164688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164654    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.164688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164667    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.164688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164679    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6104a6ef-55fc-4e00-b103-94fc61c07e34" containerName="maas-api"
Jul 23 10:03:58.164688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164685    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6104a6ef-55fc-4e00-b103-94fc61c07e34" containerName="maas-api"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164700    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164706    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164729    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerName="maas-api"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164737    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerName="maas-api"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164810    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.164821 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164817    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c9d82a-37ec-4deb-a268-4f9ac9636c6e" containerName="maas-api"
Jul 23 10:03:58.165056 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164828    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6104a6ef-55fc-4e00-b103-94fc61c07e34" containerName="maas-api"
Jul 23 10:03:58.165056 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.164965    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19323cd3-b3f1-4e49-8e8f-12a69d2def1a" containerName="istio-proxy"
Jul 23 10:03:58.169041 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.169026    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.171493 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.171469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmqp\"/\"kube-root-ca.crt\""
Jul 23 10:03:58.172514 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.172490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qcmqp\"/\"default-dockercfg-7c6dg\""
Jul 23 10:03:58.172641 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.172501    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qcmqp\"/\"openshift-service-ca.crt\""
Jul 23 10:03:58.178437 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.178413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/must-gather-jsrh6"]
Jul 23 10:03:58.257850 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.257821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2264p\" (UniqueName: \"kubernetes.io/projected/43526cbc-7332-4345-a478-da70a78a958a-kube-api-access-2264p\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.257985 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.257875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43526cbc-7332-4345-a478-da70a78a958a-must-gather-output\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.358316 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.358296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2264p\" (UniqueName: \"kubernetes.io/projected/43526cbc-7332-4345-a478-da70a78a958a-kube-api-access-2264p\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.358412 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.358335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43526cbc-7332-4345-a478-da70a78a958a-must-gather-output\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.358640 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.358625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/43526cbc-7332-4345-a478-da70a78a958a-must-gather-output\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.369503 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.369478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2264p\" (UniqueName: \"kubernetes.io/projected/43526cbc-7332-4345-a478-da70a78a958a-kube-api-access-2264p\") pod \"must-gather-jsrh6\" (UID: \"43526cbc-7332-4345-a478-da70a78a958a\") " pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.479040 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.478977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/must-gather-jsrh6"
Jul 23 10:03:58.606846 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.606820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/must-gather-jsrh6"]
Jul 23 10:03:58.608796 ip-10-0-129-47 kubenswrapper[2560]: W0723 10:03:58.608751    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43526cbc_7332_4345_a478_da70a78a958a.slice/crio-8d892853fda456c98804b0f30dc685a4f27c19f3bce9bd93259488a74b34c26c WatchSource:0}: Error finding container 8d892853fda456c98804b0f30dc685a4f27c19f3bce9bd93259488a74b34c26c: Status 404 returned error can't find the container with id 8d892853fda456c98804b0f30dc685a4f27c19f3bce9bd93259488a74b34c26c
Jul 23 10:03:58.610808 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.610789    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 10:03:58.819002 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:58.818940    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/must-gather-jsrh6" event={"ID":"43526cbc-7332-4345-a478-da70a78a958a","Type":"ContainerStarted","Data":"8d892853fda456c98804b0f30dc685a4f27c19f3bce9bd93259488a74b34c26c"}
Jul 23 10:03:59.827837 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:59.827803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/must-gather-jsrh6" event={"ID":"43526cbc-7332-4345-a478-da70a78a958a","Type":"ContainerStarted","Data":"62d321f3e0bb936c741d9ece46b3c8f72deeecdafff025ac3446304c09c97759"}
Jul 23 10:03:59.828332 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:59.827852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/must-gather-jsrh6" event={"ID":"43526cbc-7332-4345-a478-da70a78a958a","Type":"ContainerStarted","Data":"c71cee61f56a27c560843d78659f7a9ca3e8b563bf360e421821dc6b8f91208b"}
Jul 23 10:03:59.847156 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:03:59.847097    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qcmqp/must-gather-jsrh6" podStartSLOduration=1.116392063 podStartE2EDuration="1.847078668s" totalImagesPullingTime="730.686605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 10:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:03:59.84454003 +0000 UTC m=+2824.522757814" watchObservedRunningTime="2026-07-23 10:03:59.847078668 +0000 UTC m=+2824.525296451"
Jul 23 10:04:01.012880 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:01.012804    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rq4z_2e4b6b60-ef78-4bd4-b917-c3271a1fdbd5/global-pull-secret-syncer/0.log"
Jul 23 10:04:01.195651 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:01.195609    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-czvh8_2228873f-e114-44b8-80bf-34a2f2fc7932/konnectivity-agent/0.log"
Jul 23 10:04:01.257768 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:01.257722    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-47.ec2.internal_ea26f9754e040fa3e04439c1b1bcf115/haproxy/0.log"
Jul 23 10:04:07.252688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:07.252593    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7cgdk_1534f616-ae0c-4e0e-86a3-9dbed3d7e356/manager/0.log"
Jul 23 10:04:07.281668 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:07.281546    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-p5hbq_91e4a155-59b8-4e44-84c1-007a13e33cd5/kuadrant-console-plugin/0.log"
Jul 23 10:04:07.334523 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:07.334484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-lb4gb_44ce8fa2-fd76-409b-8cca-5ffe1850f3bd/registry-server/0.log"
Jul 23 10:04:07.532252 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:07.532166    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7mzw2_d83f55d6-df56-4db5-9a46-df7abbbe1c0a/manager/0.log"
Jul 23 10:04:07.683900 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:07.682604    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lhn2f_985c8d9d-ff72-4873-a70d-e29994d39ed4/manager/0.log"
Jul 23 10:04:09.622867 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.622825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-f7g8w_5cc311f4-b27d-48e0-b77e-d016f244e47f/kube-state-metrics/0.log"
Jul 23 10:04:09.644241 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.644208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-f7g8w_5cc311f4-b27d-48e0-b77e-d016f244e47f/kube-rbac-proxy-main/0.log"
Jul 23 10:04:09.666694 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.666654    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-f7g8w_5cc311f4-b27d-48e0-b77e-d016f244e47f/kube-rbac-proxy-self/0.log"
Jul 23 10:04:09.717999 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.717966    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-j8dnb_c3e86f29-9b75-4f4c-a849-f8fe32f7e83b/monitoring-plugin/0.log"
Jul 23 10:04:09.751226 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.751196    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25jw7_a8244060-d400-443d-b19e-b02262c775a9/node-exporter/0.log"
Jul 23 10:04:09.776950 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.776902    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25jw7_a8244060-d400-443d-b19e-b02262c775a9/kube-rbac-proxy/0.log"
Jul 23 10:04:09.803792 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.803745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-25jw7_a8244060-d400-443d-b19e-b02262c775a9/init-textfile/0.log"
Jul 23 10:04:09.990608 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:09.990567    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-qjsnd_7cc22e99-9137-4fe1-90bb-a1a1fe741bc3/kube-rbac-proxy-main/0.log"
Jul 23 10:04:10.013112 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.013077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-qjsnd_7cc22e99-9137-4fe1-90bb-a1a1fe741bc3/kube-rbac-proxy-self/0.log"
Jul 23 10:04:10.035507 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.035472    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-qjsnd_7cc22e99-9137-4fe1-90bb-a1a1fe741bc3/openshift-state-metrics/0.log"
Jul 23 10:04:10.067787 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.067755    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/prometheus/0.log"
Jul 23 10:04:10.089065 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.089035    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/config-reloader/0.log"
Jul 23 10:04:10.118188 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.118158    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/thanos-sidecar/0.log"
Jul 23 10:04:10.139990 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.139959    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/kube-rbac-proxy-web/0.log"
Jul 23 10:04:10.160471 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.160432    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/kube-rbac-proxy/0.log"
Jul 23 10:04:10.181018 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.180985    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/kube-rbac-proxy-thanos/0.log"
Jul 23 10:04:10.202533 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.202491    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_86d292c4-c36f-4855-b678-397d99952902/init-config-reloader/0.log"
Jul 23 10:04:10.232355 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.232317    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8lhnr_f1101cc3-0d14-489a-af1e-e38f6e3cbe67/prometheus-operator/0.log"
Jul 23 10:04:10.248289 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.248180    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8lhnr_f1101cc3-0d14-489a-af1e-e38f6e3cbe67/kube-rbac-proxy/0.log"
Jul 23 10:04:10.276548 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.276514    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-mxm6v_699488dc-2e27-4a52-bb09-36e7718ee085/prometheus-operator-admission-webhook/0.log"
Jul 23 10:04:10.383461 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.383424    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/thanos-query/0.log"
Jul 23 10:04:10.404177 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.404145    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/kube-rbac-proxy-web/0.log"
Jul 23 10:04:10.424683 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.424650    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/kube-rbac-proxy/0.log"
Jul 23 10:04:10.444600 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.444571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/prom-label-proxy/0.log"
Jul 23 10:04:10.465402 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.465373    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/kube-rbac-proxy-rules/0.log"
Jul 23 10:04:10.485295 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:10.485263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4756556-wcnzv_e7f0bc11-d4b4-416c-a62e-c45aa1463576/kube-rbac-proxy-metrics/0.log"
Jul 23 10:04:11.389593 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.389553    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"]
Jul 23 10:04:11.395429 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.395400    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.400007 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.399978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"]
Jul 23 10:04:11.513424 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.513380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-sys\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.513424 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.513420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwjp\" (UniqueName: \"kubernetes.io/projected/dcba16b6-26b0-4949-a1b0-4f3429831e65-kube-api-access-hzwjp\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.513700 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.513490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-podres\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.513700 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.513519    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-proc\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.513700 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.513563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-lib-modules\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615203 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-podres\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615395 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615221    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-proc\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615395 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-lib-modules\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615395 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-sys\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615561 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwjp\" (UniqueName: \"kubernetes.io/projected/dcba16b6-26b0-4949-a1b0-4f3429831e65-kube-api-access-hzwjp\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.615897 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-podres\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.616019 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.615976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-proc\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.616085 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.616070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-lib-modules\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.616142 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.616114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcba16b6-26b0-4949-a1b0-4f3429831e65-sys\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.623046 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.623018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-nbb7b_c22ced1f-2f2d-47e5-98e1-e70166d53f22/networking-console-plugin/0.log"
Jul 23 10:04:11.624127 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.624104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwjp\" (UniqueName: \"kubernetes.io/projected/dcba16b6-26b0-4949-a1b0-4f3429831e65-kube-api-access-hzwjp\") pod \"perf-node-gather-daemonset-l72s4\" (UID: \"dcba16b6-26b0-4949-a1b0-4f3429831e65\") " pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.709688 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.709645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:11.881366 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.881330    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"]
Jul 23 10:04:11.882465 ip-10-0-129-47 kubenswrapper[2560]: W0723 10:04:11.882434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcba16b6_26b0_4949_a1b0_4f3429831e65.slice/crio-97971ca6604574554c6219d60c58c69d9b883e97cc78234b3bc5767fa7adabe7 WatchSource:0}: Error finding container 97971ca6604574554c6219d60c58c69d9b883e97cc78234b3bc5767fa7adabe7: Status 404 returned error can't find the container with id 97971ca6604574554c6219d60c58c69d9b883e97cc78234b3bc5767fa7adabe7
Jul 23 10:04:11.951962 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:11.951885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4" event={"ID":"dcba16b6-26b0-4949-a1b0-4f3429831e65","Type":"ContainerStarted","Data":"97971ca6604574554c6219d60c58c69d9b883e97cc78234b3bc5767fa7adabe7"}
Jul 23 10:04:12.960290 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:12.960249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4" event={"ID":"dcba16b6-26b0-4949-a1b0-4f3429831e65","Type":"ContainerStarted","Data":"8a52f838e6d4d27a96ecd14d4eb426218b292f32528b0fe1728ddac035e5b50a"}
Jul 23 10:04:12.960801 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:12.960367    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:12.977503 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:12.977443    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4" podStartSLOduration=1.977425993 podStartE2EDuration="1.977425993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 10:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 10:04:12.975399578 +0000 UTC m=+2837.653617388" watchObservedRunningTime="2026-07-23 10:04:12.977425993 +0000 UTC m=+2837.655643776"
Jul 23 10:04:14.289072 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:14.289029    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cs9ms_9b3f90a5-4578-42d2-83fe-e104048cfaae/dns/0.log"
Jul 23 10:04:14.309116 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:14.309089    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cs9ms_9b3f90a5-4578-42d2-83fe-e104048cfaae/kube-rbac-proxy/0.log"
Jul 23 10:04:14.466724 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:14.466688    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7lc8_e6e5add7-e2a7-47a6-a2e9-8685a6af4dff/dns-node-resolver/0.log"
Jul 23 10:04:15.019280 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:15.019236    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c6c89897c-2zg9x_85e5b1e4-1fef-45d9-b7d6-b1a35de9a7e9/registry/0.log"
Jul 23 10:04:15.110525 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:15.110490    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xzzhs_f744bf95-832b-4e38-8967-3f794694cdbd/node-ca/0.log"
Jul 23 10:04:16.288772 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:16.288727    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-wrrg6_1bef368c-87c7-495b-87d5-3a834ebb9e15/payload-pre-processing/0.log"
Jul 23 10:04:16.332997 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:16.332953    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-x2k4v_5584fdd3-e2a9-429e-a7b2-8367834a10fa/payload-processing/0.log"
Jul 23 10:04:17.052070 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:17.052043    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jghtb_5cf121fd-4c05-4506-8496-e70b6f7e2bec/serve-healthcheck-canary/0.log"
Jul 23 10:04:17.612904 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:17.612871    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft8xt_0a6b63b1-e2b7-4cab-ba7f-94c406853555/kube-rbac-proxy/0.log"
Jul 23 10:04:17.633285 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:17.633243    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft8xt_0a6b63b1-e2b7-4cab-ba7f-94c406853555/exporter/0.log"
Jul 23 10:04:17.654936 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:17.654886    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ft8xt_0a6b63b1-e2b7-4cab-ba7f-94c406853555/extractor/0.log"
Jul 23 10:04:18.976635 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:18.976608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qcmqp/perf-node-gather-daemonset-l72s4"
Jul 23 10:04:20.408032 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:20.407994    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c9f8fb88-8m69g_27950d6c-29fa-47e0-b465-8507f8a016c6/manager/0.log"
Jul 23 10:04:20.430472 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:20.430443    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-8vjr4_bd2e00f1-3288-43a1-b3ee-c11dd12442db/server/0.log"
Jul 23 10:04:20.625753 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:20.625708    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-2765v_7051f641-18e2-44e6-80e3-b581c48b5066/manager/0.log"
Jul 23 10:04:20.645809 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:20.645770    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64d6699bc9-2765v_7051f641-18e2-44e6-80e3-b581c48b5066/copy-manifests/0.log"
Jul 23 10:04:22.084983 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:22.084950    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cb78dd96b-hrk74_3cfe5411-dc5f-4f5a-a663-c96a96515a5a/manager/0.log"
Jul 23 10:04:22.132196 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:22.132159    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-wrrg6_1bef368c-87c7-495b-87d5-3a834ebb9e15/payload-pre-processing/0.log"
Jul 23 10:04:22.160734 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:22.160616    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-x2k4v_5584fdd3-e2a9-429e-a7b2-8367834a10fa/payload-processing/0.log"
Jul 23 10:04:23.700655 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:23.700600    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-wrrg6_1bef368c-87c7-495b-87d5-3a834ebb9e15/payload-pre-processing/0.log"
Jul 23 10:04:23.736832 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:23.736773    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-x2k4v_5584fdd3-e2a9-429e-a7b2-8367834a10fa/payload-processing/0.log"
Jul 23 10:04:28.741848 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:28.741811    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2l9bs_6656f6a5-1eb8-4d38-8ee7-9a57e4fb4300/kube-multus/0.log"
Jul 23 10:04:29.098670 ip-10-0-129-47 kubenswrapper[2560]: I0723 10:04:29.098578    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rl4q7_460e979e-3746-4b38-8768-779560aba737/kube-multus-additional-cni-plugins/0.log"